The novel opens in a Confederate military hospital near Raleigh, North Carolina where Inman is recovering from battle wounds during the American Civil War. The soldier is tired of fighting for a cause he never believed in. After considering the advice from a blind man and moved by the death of the man in the bed next to him, he decides one nightfall to slip out of the hospital in order to return to his home at Cold Mountain, North Carolina.
At Cold Mountain, Ada's father soon dies. The farm, named Black Cove, that the genteel city-bred Ada lives on is soon reduced to a state of disrepair. But she is saved from destitution by a resourceful-but-homeless young woman named Ruby, who soon moves in with her. Together they soon clean the place up and return it to productivity. Ruby also teaches Ada how to survive in these very different times.
Inman soon becomes aware of the Confederate Home Guard who hunt down military deserters from the Confederacy. He meets a preacher called Veasey, whom he catches in the act of attempting to murder his pregnant lover. After Inman dissuades him, they travel together. They butcher a dead cow that had fallen into a creek and the cow's owner, Junior, gives them away to the Home Guard. They are put into a group of other captured prisoners, and march for days before the Home Guard decides to simply shoot them because they are "too much trouble". Veasey steps forward to try to stop them and is killed. Inman survives when he takes a graze from a bullet that has already gone through Veasey and they think he is dead. They dig a shoddy mass grave and Inman pulls himself out, helped in part by some passing wild pigs. He cannot bury Veasey so he turns him face down and continues on.
Inman's journey is rough. He faces hunger and an attempted armed robbery at a rural tavern even though he carries a LeMat revolver for protection. Occasionally he is helped and sheltered by civilians who want nothing to do with the war. Through cunning ingenuity he helps one of them track and recover a hog, her only possession and source of food for the winter which had just been seized from her by Union soldiers. He is also helped by a woman who owns goats, who gives him advice and medicines to finally heal his wounds.
Ruby's father, Stobrod, is caught stealing corn at Ada's farm. Ruby reveals he was a deadbeat who abused and neglected her when she was very young; he is also a Confederate deserter. Nevertheless Ruby grudgingly feeds him. Soon he returns another day with a simple-minded friend named Pangle. Together they entertain everyone by playing the fiddle and banjo. However the Home Guard, led by the sadistic Captain Teague, eventually track them down and shoot them. A third companion, referred to as "Georgia", escapes the killing and goes off to alert Ada and Ruby. The two women ride and find Stobrod barely alive. Ada and Ruby pitch camp to give him a place to recover.
After Inman arrives at Black Cove to find it empty, he sets out to find Ada on the mountain. Unexpectedly he soon encounters her out hunting wild turkeys. Both have changed so greatly in their appearance and demeanor since they parted that it is some moments before they recognize one another. Inman takes up camp with Ada and Ruby. Ruby is afraid Ada will dismiss her now she has a husband, and Ada reassures her that she needs her as a friend and for her ideas and help. Ruby gives the pair her blessing. Later Ada and Inman make love. They happily begin to imagine the life they will have together at Black Cove and make plans for their future.
However, as the party begins the trek back to the farm they encounter the Home Guard. A shoot-out commences in which Inman kills all the members of the Home Guard except for a 17-year-old Birch, Teague's vicious protégé. Inman eventually corners the boy against a rock ledge where he is reluctant to shoot him down in cold blood. However, after attempts fail to convince Birch to lay down his arms and leave, the boy shoots and kills Inman.
Ada is left a pregnant widow. She raises her daughter at Black Cove where she lives with Ruby and Stobrod. Ruby has married the boy from Georgia, named Reid, and has three children.