On a visit to Austerlitz in the late 1950s, Oliver met photographer Molly Malone Cook, who would become her partner for over forty years.[4] In Our World, a book of Cook's photos and journal excerpts Oliver compiled after Cook's death, Oliver writes, "I took one look [at Cook] and fell, hook and tumble." Cook was Oliver's literary agent. They made their home largely in Provincetown, Massachusetts, where they lived until Cook's death in 2005, and where Oliver continued to live[10] until relocating to Florida.[15] Of Provincetown, she recalled, "I too fell in love with the town, that marvelous convergence of land and water; Mediterranean light; fishermen who made their living by hard and difficult work from frighteningly small boats; and, both residents and sometime visitors, the many artists and writers.[...] M. and I decided to stay."[4]
Oliver valued her privacy and gave very few interviews, saying she preferred for her writing to speak for itself.[6]
