Gogol states in his author's note that Viy, the King of the Gnomes, was an actual character from Ukrainian folklore. This was merely a literary device. In reality Gogol probably never heard of Viy at all. No discovery has been made of the folklore source of Viy, and as such it remains a part of Gogol's imagination. However, some scholars believe that the conception of Viy may have been at least partially based on old folk tradition surrounding St. Cassian the Unmerciful, who was said in some tales to have eyebrows that descended to his knees and which were raised only on Leap Year. In some examples, he has been depicted as almost a demon in the sky, so it's likely Gogol at least heard about the character and designed Viy on his various forms.
The demons summoned into the church come from the Slavic superstitions of "midnight dead". Evil people, it was believed, automatically became Devil's subjects upon death. Earth would not hold them so that every night they would crawl out of their graves and torment the living. In the story, the demons have "black earth" clung to them, as if they crawled out of the ground.
The water sprite (Rusalka) seen by Khoma during his night ride bears relation to the "midnight dead". It was widely believed, in Russian and Ukraine, that rusalki were spirits of unbaptized children or drowned maidens, who were in league with the Devil. They were known to drown their victims or tickle them to death. They were described as beautiful, and deadly, and bear relation to the young version of the witch, and Gogol's frequent portrayal of women as beautiful yet evil.
Incantations, exorcism, and the magic circle come from Ukrainian beliefs of protection from evil forces. The circle relates to "chur", a magical boundary that evil cannot cross. Even though Khoma died from fear, the creatures could not touch him.
Additionally, the final notion that Khoma died only because he let fear win over him appears to stem from John of Damascus, who said "... all evil and impure passions have been conceived by [evil spirits] and they have been permitted to visit attacks upon man. But they are unable to force anyone, for it is in our power either to accept the visitation or not."