Poe's inspiration for the story may be based upon events of the Hezekiah Usher House, which was located on the Usher estate that is now a three-block area in modern Boston, Massachusetts adjacent to Boston Common and bound by Tremont Street to the northwest, Washington Street to the southeast, Avery Street to the south and Winter Street to the north. The house was constructed in 1684 and either torn down or relocated in 1830. Other sources indicate that a sailor and the young wife of the older owner were caught and entombed in their trysting spot by her husband. When the Usher House was torn down in 1830, two bodies were found embraced in a cavity in the cellar.
Another source of inspiration may be from an actual couple by the name Mr. and Mrs. Luke Usher, the friends and fellow actors of his mother Eliza Poe. The couple took care of Eliza's three children (including Poe) during her time of illness and eventual death.
German writer E.T.A. Hoffmann, who was a role model and inspiration for Poe, published the story "Das Majorat" in 1819. There are many similarities between the two stories, like the breaking in two of a house, eerie sounds in the night, the story within a story and the house owner's being called Roderich. Because Poe was familiar with Hoffmann's works, he knew the story and cleverly drew from it using the elements for his own purposes.
Another German author, Heinrich Clauren's, 1812 story The Robber's Castle, as translated into English by John Hardman and published in Blackwood's Magazine in 1828 as "The Robber's Tower", may have served as an inspiration according to Arno Schmidt and Thomas Hansen. As well as common elements, such as a young woman with a fear of premature burial interred in a sepulchre directly beneath the protagonist's chamber, stringed instruments and the living twin of the buried girl, Diane Hoeveler identifies textual evidence of Poe's use of the story, and concludes that the inclusion of Vigiliae Mortuorum secundum Chorum Ecclesiae Maguntinae (Vigils for the Dead according to the Use of the Church of Mainz) is drawn from the use of a similarly obscure book in "The Robber's Tower".
The theme of the crumbling, haunted castle is a key feature of Horace Walpole's Castle of Otranto (1764), which largely contributed in defining the Gothic genre.