Chapter 8 - Rotational Motion and Equilibrium - Learning Path Questions and Exercises - Multiple Choice Questions - Page 301: 7

(a) its potential energy will decrease

When an object in unstable equilibrium is displaced slightly, its potential energy will decrease. Because when the object is in unstable equilibrium it is already in highest potential energy state possible. The height of CG is maximum. When it is displaced slightly, the height decreases, as a result the potential energy decreases. Hence, (a) is the answer.