Why doesn’t Chillingworth assert his rights as Hester’s husband?

Chillingworth doesn't want his relationship with Hester revealed. He has no wish for people to know that his wife has been unfaithful, made a fool of him, nor that his wife had a child with his lover. He also has no desire to explain to anyone why he deserted his wife and left her on his own while he lived with the Indians.