What is Parris afraid will happen if the townspeople associate witchcraft with his home?

Parris is afraid that those who don't like him.... his enemies, will use the association to depose him from his position in the church.