Answer
(a) $x = \frac{2A}{B}$
(b) $x = \frac{2A}{B}$ is a stable equilibrium.
Work Step by Step
(a) We can find the positions where $F = 0$:
$F = -\frac{dU}{dx} = 0$
$-(-\frac{2A}{x^3}+\frac{B}{x^2}) = 0$
$\frac{B}{x^2} = \frac{2A}{x^3}$
$Bx = 2A$
$x = \frac{2A}{B}$
(b) We can find an expression for $\frac{d^2U}{dx^2}$:
$U(x) = \frac{A}{x^2}-\frac{B}{x}$
$\frac{dU}{dx} = -\frac{2A}{x^3}+\frac{B}{x^2}$
$\frac{d^2U}{dx^2} = \frac{6A}{x^4}-\frac{2B}{x^3}$
When $x = \frac{2A}{B}$:
$\frac{d^2U}{dx^2} = \frac{6A}{x^4}-\frac{2B}{x^3}$
$\frac{d^2U}{dx^2} = \frac{3B^4}{8A^3}-\frac{B^4}{4A^3}$
$\frac{d^2U}{dx^2} = \frac{B^4}{8A^3}$
When $x = \frac{2A}{B},$ then $\frac{d^2U}{dx^2} \gt 0$, so this is a local minimum.
$x = \frac{2A}{B}$ is a stable equilibrium.