What is the ''wall of stone'' between Thoreau his "townsmen"?

The wall of stone refers to the difference between how Thoreau saw the tax and government differently than his fellow citizens. It was like they were separated by a wall of stone:“I saw that, if there was a wall of stone between me and my townsmen, there was a still more difficult one to break through, before they could get to be as free as I was."