Born in Stepney in London's East End, Walter Pater was the second son of Richard Glode Pater, a physician who had moved to London in the early 19th century to practice medicine among the poor. Dr Pater died while Walter was an infant and the family moved to Enfield, London. Walter attended Enfield Grammar School and was individually tutored by the headmaster.
In 1853, he was sent to The King's School, Canterbury, where the beauty of the cathedral made an impression that would remain with him all his life. He was fourteen when his mother, Maria Pater, died in 1854. As a schoolboy Pater read John Ruskin's Modern Painters, which helped inspire his lifelong attraction to the study of art and gave him a taste for well-crafted prose. He gained a school exhibition, with which he proceeded in 1858 to Queen's College, Oxford.