Answer
Inflating the balloon at sea level requires more work.
Work Step by Step
This is the case of an expansion against an external pressure. The pressure you are fighting at sea level is higher than atop a mountain, so you do more work in expanding the balloon at sea level.