Alice Duer was born in New York City on July 28, 1874 into a wealthy family. She was the daughter of James Gore King Duer and Elizabeth Wilson Meads. Elizabeth was the daughter of Orlando Meads of Albany, New York. Her great grandfather was William Alexander Duer, who was the president of Columbia College, 1829–1842. Her great great grandfather, was William Duer, an American lawyer, developer, and speculator from New York City. He had served in the Continental Congress and the convention that framed the New York Constitution. In 1778, he signed the United States Articles of Confederation. Her great great great grandfather was William Alexander, who claimed the disputed title of Earl of Stirling, and was an American Major-General during the American Revolutionary War.
She was also a descendant of Senator Rufus King, who was an American lawyer, politician, and diplomat. He was a delegate for Massachusetts to the Continental Congress. He also attended the Constitutional Convention and was one of the signers of the United States Constitution on September 17, 1787, in Philadelphia, Pennsylvania. He represented New York in the United States Senate, served as Minister to Britain, and was the Federalist candidate for both Vice President (1804, 1808) and President of the United States (1816)
At the time of her entrance into society, her family lost most of its fortune. She entered Barnard College in 1895 studying mathematics and astronomy. She helped to pay for her studies by selling novels and short essays. She and her sister Caroline King Duer published a joint book of poems. Alice graduated in June 1899.
On October 5, 1899, she married Henry Wise Miller at Grace Church Chapel in New York City. He was born in 1877, the son of Lt. Commander Jacob Miller, in Nice, France, where his father was serving with the U.S. Navy. He was an 1892 graduate of Harvard University. They left for Costa Rica, where he attempted to develop rubber cultivation. This venture eventually failed; in 1903, she, her husband and young son returned to New York, where they lived in difficulty for some time, he working in the Stock Exchange, she teaching, which she hated. After a time, her husband earned more and she was able to dedicate her working time entirely to writing.