The novel centers on a wealthy family living in New Rochelle, New York, simply named "Father," "Mother," "Mother's Younger Brother," and "Grandfather." Their young son, who narrates the novel from the perspective of an adult reminiscing about the people and events of his childhood, is not named at all. The family business is the manufacture of flags and fireworks, an easy source of wealth due to the national enthusiasm for patriotic displays. Father joins the first expedition to the North Pole, and his return sees a change in the sexual politics of his relationship with his wife, who has experienced a taste of independence in his absence. Younger Brother is a genius at explosives and fireworks, but is an insecure, unhappy character who chases after love and excitement. He becomes obsessed with the notorious socialite Evelyn Nesbit, stalking her through the city and eventually embarking on a brief, unsatisfactory affair with her that leaves him even more isolated.
Into this insecure setup comes first an abandoned black child, then his severely depressed mother Sarah. Coalhouse Walker, the child's father, visits regularly to win Sarah's affections. A professional musician, well dressed and well spoken, he gains the family's respect and overcomes their prejudice initially by playing ragtime music on their badly tuned piano. Things go well until he is humiliated by a racist fire crew, lead by Will Conklin, who vandalize his Model T Ford. He begins a single-minded pursuit of legal action but discovers he has no case due to the prejudices of the system. Sarah is killed in an attempt to aid him, and Coalhouse uses the money he was saving for their wedding to pay for an extravagant funeral. Having exhausted legal resources, Coalhouse begins killing firemen and bombing firehouses to force the city to meet his demands: that his Model T be restored to its original condition, and Conklin be turned over to him for justice. Mother unofficially adopts Sarah and Coalhouse's neglected child over Father's objections, putting strain on their marriage. With a group of angry young men, all of whom come to refer to themselves as 'Coalhouse Walker', Coalhouse continues his vigilante campaign, and is eventually joined by Younger Brother, who brings with him his knowledge of explosives. Coalhouse and his gang storm the Morgan Library, taking the priceless collection within hostage and wiring the building with dynamite. Father is drawn into the escalating conflict as a mediator, as is Booker T. Washington. Coalhouse eventually agrees to exchange Conklin's life for safe passage for his men, who leave in his restored Model T. Coalhouse is then shot as he surrenders to the authorities.
Interwoven with this story is a depiction of life in the tenement slums of New York city, focused on the Eastern European socialist immigrant Tateh, who struggles to support himself and his daughter after driving her mother off for accepting money for sex from her employer. The girl's beauty attracts the attention of Evelyn Nesbit, who provides financial support. When Tateh learns her identity, however, it drives him to take his daughter out of the city. Tateh is a talented artist, and earns a living cutting out novelty paper silhouettes on the street, later working in a factory. After a successful factory workers' strike changes little about the workers' lives, he becomes disillusioned and moves away from his socialist roots to become an entrepreneur. He makes and sells moving picture books to a novelty toy company, progressing into a pioneer animation in the moving picture industry. Tateh subsequently becomes wealthy and styles himself as 'the baron', in order to move more easily through high society. He meets, and falls in love with Mother, who eventually marries him after Father is killed in the sinking of the RMS Lusitania. They adopt each other's children, as well as Coalhouse's son, and move to California.
Mixed into the interwoven stories are subplots following prominent figures of the day, including JP Morgan, Harry Houdini, Henry Ford, Siegmund Freud and Emma Goldman.