He was born near Kingston, Canada West (now incorporated into Ontario), the second son of Catharine Ann Grant and the Rev. Joseph Antisell Allen, a Protestant minister from Dublin, Ireland. His mother was a daughter of the fifth Baron of Longueuil. He was educated at home until, at age 13, he and his parents moved to the United States, then France and finally the United Kingdom. He was educated at King Edward's School in Birmingham and Merton College in Oxford, both in the United Kingdom. After graduation, Allen studied in France, taught at Brighton College in 1870–71 and in his mid-twenties became a professor at Queen's College, a black college in Jamaica.