Answer
a) The net force on the mass $M$ is zero at $r = \frac{\sqrt{3}-1}{2}$ and $r = -\frac{\sqrt{3}+1}{2}$.
b) i) It is an unstable equilibrium at both points.
ii) It is a stable equilibrium at both points.
Work Step by Step
a) Let the required position of the mass $M$ be at a distance $r$ from the body of mass $m$ as shown in the figure. The distance between the bodies of mass $m$ and $3m$ is $1\mathrm{m}$. The net force on the body of mass $M$ is
$F = \frac{3GMm}{(1-r)^2} - \frac{GMm}{r^2}$
since the mass $3m$ exerts a force in the $+x$ direction and the mass $m$ exerts a force in the $-x$ direction.
If the net force is zero,
$ \frac{3GMm}{(1-r)^2} - \frac{GMm}{r^2}=0$
$\implies \sqrt{3}r = \pm (1-r)$
$\implies r = \frac{1}{1+\sqrt{3}}, \frac{-1}{\sqrt{3}-1}$
$\implies r = \frac{\sqrt{3}-1}{2},-\frac{\sqrt{3}+1}{2}$
Thus there are two points where the net force is zero, one between the two masses at $r= \frac{\sqrt{3}-1}{2}$ from the mass $m$, and the other to the left of the mass $m$ at a position $r = -\frac{\sqrt{3}+1}{2}$.
b) i) At the point $r = \frac{\sqrt{3}-1}{2}$, when the mass $M$ is pushed towards the mass $m$, it is less attracted by $3m$ and more attracted by $m$ and it thus falls towards the mass $m$. The same argument holds for pushing towards the mass $3m$. This point is thus in unstable equilibrium.
At the point $r = -\frac{\sqrt{3}+1}{2}$, when the mass is pushed towards the mass $m$, the pull of both $m$ and $3m$ increases and thus the mass $M$ falls towards $m$. This is again thus unstable equilibrium.
ii) At both points $r = \frac{\sqrt{3}-1}{2}$ and $r = -\frac{\sqrt{3}+1}{2}$, when the mass M is pushed slightly in a direction perpendicular to the line joining $m$ and $3m$, the force of gravity pull the mass $M$ back to its equilibrium position but due to the inertia, it moves in the opposite direction by an equal amount, where it is again pulled back to its equilibrium position by gravity and this motion repeats. The mass $M$ executes simple harmonic motion for small displacements in the direction perpendicular to the line joining $m$ and $3m$. This is stable equilibrium.
