The novel opens in a Confederate military hospital in Raleigh, North Carolina, where Inman is recovering from battle wounds during the American Civil War. The soldier is tired of fighting for a cause he never believed in. After considering the advice of a blind man, and moved by the death of the man in the bed next to him, he decides one night to slip out of the hospital and return home to Cold Mountain, North Carolina.
At Cold Mountain, Ada's father dies. The farm, named Black Cove, that the genteel, city-bred Ada lives on is soon reduced to a state of disrepair. But she is saved from destitution by a resourceful but homeless young woman named Ruby, who moves in with her. Together, they clean the place up and return it to productivity. Ruby also teaches Ada how to survive in those difficult times, while Ada shares her knowledge of literature with Ruby.
Inman soon becomes aware of the Confederate Home Guard, who hunt down military deserters from the Confederacy. He meets a preacher called Veasey, whom he catches in the act of attempting to murder a woman he has impregnated. After Inman dissuades him, they travel together. They butcher a dead bull that had fallen into a creek and the bull's owner, Junior, gives them away to the Home Guard. They are put into a group of other captured prisoners and march for days before the Home Guard decides to simply shoot them because they are "too much trouble." Veasey steps forward to try to stop them and is killed. Inman is grazed by a bullet that passed through Veasey and is thought to be dead. The Guardsmen dig a shoddy mass grave and Inman pulls himself out, helped in part by some passing wild pigs. He cannot bury Veasey, so he turns him face down and continues on.
Inman's journey is rough. He faces hunger and an attempted armed robbery at a rural tavern, even though he carries a LeMat revolver for protection. Occasionally, he is helped and sheltered by civilians who want nothing to do with the war. Through cunning ingenuity, he helps one of them track and recover a hog, her only possession and source of food for the winter, which had just been seized by Union soldiers. He is also helped by a woman who owns goats, who gives him advice and medicines to finally heal his wounds.
Ruby's father, Stobrod, is caught stealing corn at Ada's farm. Ruby reveals he was a deadbeat who neglected and abandoned her when she was very young; he is also a Confederate deserter. Nevertheless, Ruby grudgingly feeds him. Soon he returns another day with a simple-minded friend named Pangle. Together they entertain everyone by playing the fiddle and banjo. However, the Home Guard, led by the sadistic Captain Teague, eventually tracks them down and shoots them. A third companion, referred to as "Georgia," escapes the killing and goes off to alert Ada and Ruby. The two women ride and find Stobrod barely alive. Ada and Ruby pitch camp to give him a place to recover.
After Inman arrives at Black Cove to find it empty, he sets out to find Ada on the mountain. Unexpectedly he soon encounters her out hunting wild turkeys. Both have changed so greatly in their appearance and demeanor since they parted that it is some moments before they recognize one another. Inman takes up camp with Ada and Ruby. Ruby is afraid Ada will dismiss her now she has a husband, and Ada reassures her that she needs her as a friend and for her ideas and help. Ruby gives the pair her blessing. Later Ada and Inman make love. They happily begin to imagine the life they will have together at Black Cove and make plans for their future.
However, as the party begins the trek back to the farm, they encounter the Home Guard. A shootout commences in which Inman kills all the members of the Home Guard except for 17-year-old Birch, Teague's vicious protégé. Inman eventually corners the boy against a rock ledge but is reluctant to shoot him down in cold blood. However, after attempts fail to convince Birch to lay down his arms and leave, the boy shoots and kills Inman.
Ada is left a pregnant widow. She raises her daughter at Black Cove, where she lives with Ruby, who got married with "Georgia" and has three sons, and Stobrod.
