how did the average person view weaves,travelers, and those who were clever?

The best answer to your question is probably in the first paragraph of the first chapter:

"In this way it came to pass that those scattered linen-weavers -- emigrants from the town into the country -- were to the last regarded as aliens by their rustic neighbours, and usually contracted the eccentric habits which belong to a state of loneliness."