Why does Chillingworth believe he has a double reason for punishing Dimmesdale?

Chillingworth believes that sinners should be outed and punished because it's the law........ his second reason is purely revenge. He wants to punish the man for his former relationship with his wife and the child they created.