Architect Norton Juster was living in his hometown of Brooklyn, after three years in the navy. In June 1960, he gained a $5,000 grant from the Ford Foundation to write a children's book about cities. Juster argued that the young baby boomers would soon have responsibility for the cities, and many lived in the suburbs and did not know them. In his proposal, he said he wanted "to stimulate and heighten perception – to help children notice and appreciate the visual world around them – to help excite them and shape their interest in an environment they will eventually reshape." Beginning with great enthusiasm, he ground to a halt with too many notes and too little progress. He took a weekend break with friends at Fire Island, and came back determined to put aside the cities book and seek inspiration in another writing project.
Milo continued to think of all sorts of things; of the many detours and wrong turns that were so easy to take, of how fine it was to be moving and, most of all, of how much could be accomplished with just a little thought.The Phantom Tollbooth (after exiting the Doldrums) 
Juster's guilt over his lack of progress on the cities book had led him to write pieces of stories about a little boy named Milo, which he began to develop into a book. Juster quit his job so that he could work on the book. His imagination fired by a boy who approached him on the street and with whom he discussed the nature of infinity, Juster wanted to finish the story about "a boy who asked too many questions" before returning to the book on cities. Juster shared his house in Brooklyn Heights with cartoonist Jules Feiffer whose bedroom was immediately below, and who could hear him pacing in the night. Feiffer was surprised to learn that his friend's insomnia was not caused by the cities book, but by a book about a boy. Juster showed Feiffer the draft to date, and, unbidden, the artist began sketching illustrations. Feiffer knew Judy Sheftel, who put deals together in the publishing trade and was his future bride. Sheftel got Jason Epstein, an innovative editor at Random House with a deep appreciation for children's literature, to agree to review the manuscript. Some at Random House considered the book's vocabulary too difficult: at the time, educators advised against children's literature containing words the target audience did not already know, fearing the unfamiliar would discourage young learners. Based on seven chapters of manuscript, plus a three-page outline of the rest of the story, Epstein bought the book.
Since Juster did the cooking for the housemates, if Feiffer wanted to eat, he had to do the drawings. Feiffer quickly realized the book would require illustrations of the type and quality that John Tenniel had created for Lewis Carroll's Alice's Adventures in Wonderland and although a nationally-known artist, doubted his competence to do the text justice. Feiffer considers the double-spread illustration of demons late in the book to be a success and one of his favorites. It differs from his usual style (which would involve a white background), and instead uses Gustave Doré's drawings as an inspiration.
It became a game, with Feiffer trying to draw things the way he wanted, and Juster trying to describe things that were impossible to sketch. These included the Triple Demons of Compromise—one short and fat, one tall and thin, and the third exactly like the first two. Feiffer got his revenge by depicting the author as the Whether Man, clad in a toga.
Repeated edits altered the protagonist's name (originally Tony), removed his parents entirely from the book, and deleted text attempting to describe how the tollbooth package had been delivered. Milo's age was removed from the text—early drafts have him aged eight or nine—as Juster decided not to state it, lest potential readers decide they were too old to care.