Why is the Scaffold the only place that the escape could take place?

The scaffold is not so much a physical escape more than a place for emotional release. Dimmesdale climbs the scaffold at night to confess his sins in the darkness. This is a cathartic experience for Dimmesdale even though most of the town is asleep.