Answer
See the detailed answer below.
Work Step by Step
a) We can see, from the figure below, that the $y$-component of the gravitational potential between $dm$'s and $m$ are equal and are in opposite directions so they cancel each other.
The same thing happens for the unseen axis $z$-axis (perpendicular to the page), the $z$-components are also having the same magnitude and opposite directions.
This means that the net gravitational potential energy between the ring $M$ and the mass $m$ is in the positive $x$-direction.
Since the mass is uniformly distributed along the ring, its density is given by
$$\rho=\dfrac{M}{V}=\dfrac{M}{AL}$$
where $L$ here is the circumference of the ring, so $L=2\pi R$
$$\rho =\dfrac{M}{2\pi R A}$$
Now the small piece of mass $dm$ has a length of $dL$ and its density is given by
$$\rho=\dfrac{dm}{dV}=\dfrac{dm}{AdL}$$
From the last two formulas,
$$\dfrac{dm}{ \color{red}{\bf\not}AdL}=\dfrac{M}{2\pi R \color{red}{\bf\not}A}$$
$$dm=\dfrac{MdL}{2\pi R}\tag 1$$
So the gravitational potential between the ring and the mass is given by
$$dU=\dfrac{-Gm dm}{r}$$
where $r$ is the hypotenuse and is given by $r=\sqrt{x^2+R^2}$
$$dU=\dfrac{-Gm dm}{\sqrt{x^2+R^2}}$$
Plugging from (1);
$$dU=\dfrac{-Gm }{\sqrt{x^2+R^2}}\dfrac{MdL}{2\pi R}$$
Taking the integral for both sides;
$$\int_0^UdU=\int\dfrac{-Gm }{\sqrt{x^2+R^2 } }\dfrac{MdL}{2\pi R}$$
Noting that $R$ and $x$ are constants.
$$ U=\dfrac{-Gm }{\sqrt{x^2+R^2 } }\dfrac{M}{2\pi R}\int\limits_{\rm ring}dL$$
$$ U=\dfrac{-Gm }{\sqrt{x^2+R^2 } }\dfrac{M}{2\pi R}L\bigg|_0^L$$
where $L$ is the circumference of the ring and is given by $2\pi R$
$$ U=\dfrac{-Gm }{\sqrt{x^2+R^2 } }\dfrac{M}{2\pi R} 2\pi R$$
Thus,
$$ \boxed{U=\dfrac{-Gm M}{\sqrt{x^2+R^2 } }} $$
---
b) We know that the force is given by
$$F=\dfrac{-dU}{dx}$$
Plugging from the boxed formula above;
$$F=\dfrac{-d}{dx}\left(\dfrac{-Gm M}{\sqrt{x^2+R^2 } }\right)$$
$$F=GMm\cdot \dfrac{d}{dx} (x^2+R^2 )^{-\frac{1}{2}} $$
$$F=GMm\cdot \left(\dfrac{-1}{2} (2x)(x^2+R^2 )^{-\frac{3}{2}} \right)$$
$$ \boxed{F=\dfrac{Gm Mx}{(x^2+R^2 )^{\frac{3}{2}} }} $$