In 1918 in memory of his brother and father, he began building the Phillips Collection, which became the first museum in the United States devoted to modern art. He married artist Marjorie Acker, and she, while continuing her art career, became Associate Director of the museum, working closely for the remainder of her life in developing the collection.