A what moment in the past does the physician say he might "have beheld the bale-fire of that scarlet letter"?

From the moment they married;

"Nay, from the moment when we came down the old church-steps together, a married pair, I might have beheld the bale-fire of that scarlet letter blazing at the end of our path!"