Gulick was born April 10, 1860 in Ebon Atoll, Marshall Islands. His father was missionary Luther Halsey Gulick Sr. (1828–1891), and mother was Louisa Mitchell (Lewis) Gulick (1830–1893). He was the brother of Luther Halsey Gulick, Jr. and grandson of missionary Peter Johnson Gulick (1796–1877). He graduated from Oakland High School in 1879. He received an A.B. degree from Dartmouth College with his brother Edward Leeds Gulick in 1883, an A.M. degree in 1886 and a D.D. degree in 1903. He also held D.D. degrees from Yale and Oberlin College.
He was ordained a Congregational minister in 1886, and then was a supply minister at the Willoughby Avenue Mission, Brooklyn. He married Clara May Fisher (1860?–1941) on November 7, 1887.