Why should Hester's marriage be a failure?

Hester and Chillingworth never loved each other. Chillingworth also left her alone with no husband to fend for herself in a place far away from home. He believes that his age and the lack of love between them is something that would have led her to adultery regardless of whether he'd been there or not.

"Nay, from the moment when we came down the old church-steps together, a married pair, I might have beheld the bale-fire of that scarlet letter blazing at the end of our path!’"