Answer
A ratio of $~~\frac{m}{M} = \frac{1}{2}~~$ gives the least gravitational potential energy for the system.
Work Step by Step
The two masses are $~~m~~$ and $~~M-m$
Let $r$ be the distance between the two masses.
We can write an expression for the gravitational potential energy of the system:
$U = -\frac{G(M-m)~m}{r^2}$
Since $U$ is a negative value, to minimize the gravitational potential energy of the system, we need to maximize the value of the product: $~~P = (M-m)\times m$
$P = (M-m)\times m$
$P = Mm-m^2$
$\frac{dP}{dm} = M-2m = 0$
$M = 2m$
$m = \frac{M}{2}$
We can find the ratio of $\frac{m}{M}$:
$\frac{m}{M} = \frac{\frac{M}{2}}{M} = \frac{1}{2}$
A ratio of $~~\frac{m}{M} = \frac{1}{2}~~$ gives the least gravitational potential energy for the system.
