Where did Dimmesdale observe his vigil?

Dimmesdale goes to the scaffold where Hester Prynne had lived through her public ordeal. He stands there, but there is no audience to witness his shame.