The novel is set in the mid-19th century, although it includes glimpses into the history of the house, which was built in the late 17th century. The house of the title is a gloomy New England mansion, haunted since its construction by fraudulent dealings, accusations of witchcraft, and sudden death. The current resident, the dignified but desperately poor Hepzibah Pyncheon, opens a shop in a side room to support her brother Clifford, who is about to leave prison after serving thirty years for murder. She refuses all assistance from her wealthy but unpleasant cousin, Judge Jaffrey Pyncheon. A distant relative, the lively and pretty young Phoebe, turns up and quickly becomes invaluable, charming customers and rousing Clifford from depression. A delicate romance grows between Phoebe and the mysterious attic lodger Holgrave, who is writing a history of the Pyncheon family.
In between, certain glimpses are provided at the history of the house and the Pyncheon family. The house was built on ground wrongfully seized from its rightful owner, Matthew Maule, by Colonel Pyncheon, the founder of the Massachusetts branch of the family, when the former was accused of practicing witchcraft and executed. According to legend, upon his death hour Maule laid a curse upon the Pyncheon family; and indeed during the house warming festivities, Colonel Pyncheon was found dead in his armchair. (It is, however, left ambiguous whether he actually died from the curse or a family-inherent disease.) His portrait remains in the house as an everlasting reminder of its dark past and the presence of the curse in the spirit of the house and its inhabitants.
Phoebe returns to her country home for a brief visit, but plans to return soon. Just before she leaves, Clifford stands at the large arched window above the stairs and has a sudden urge to jump upon viewing the mass of humanity passing before him and thinking of his youth lost to prison. This incident, coupled with Phoebe's departure — she was the only happy and beautiful thing in the home for the depressed Clifford to dwell on — sends him into a bed-ridden state.
Judge Pyncheon arrives at the house one day and threatens to have Clifford committed to an insane asylum if he does not disclose information regarding mystical "eastern lands" of Maine that the family is rumored to own, the deed to which has been lost. Before Clifford can be brought before the Judge (which, it is implied, would completely destroy Clifford's sanity), the Judge mysteriously dies while sitting in the same chair that Colonel Pyncheon had died in. Hepzibah and Clifford escape on a train (then a very new form of transport) after the Judge dies, but soon return after the first excitement and thrill of new freedom is gone and they are driven back to the house.
The next day, upon Phoebe's return, the Judge's body is discovered, and the townsfolk begin to murmur about Hepzibah and Clifford's sudden disappearance. To Phoebe's relief, Hepzibah and Clifford return shortly. Events from past and present throw light on the circumstances which sent Clifford to prison, proving his innocence: He was framed for the death of his uncle by Jaffrey, who was desperately looking for the legendary deed around the house at that time. Holgrave is discovered to be a descendant of Maule but bears the Pyncheon family no ill will, mostly due to his feelings for Phoebe; and the deed is discovered hidden behind the old Colonel's portrait, although it has long since become worthless as the land is already settled. The romance ends with the characters leaving the old house to start a new life in the countryside, free of the burdens of the past.