In The Scarlet Letter, why does the narrator lose his job in the custom house?

The narrator lost his job because of a change in the political arena. The Whigs were no longer a majority, as the Democrats had taken over. He was replaced in his post because of his political affiliations.