Answer
(a) $$U=-\dfrac{GmM}{\sqrt{x^2+a^2}}.$$
(b) see work step by step
(c)$$F_x=\dfrac{-GmMx}{(x^2+a^2)^{3/2}}$$
(d) see work step by step
(e) If $x=0$ then $U=-\dfrac{GMm}{a}$ and $F_x=0$.
Work Step by Step
First of all, we have to find the potential due to a small segment of the ring and then integrate over the entire ring to find total U. Let's divide the ring up into small segments $dM$ (see picture).
(a) the gravitational potential energy of $dM$ and $m$ is
$$dU=-\dfrac{GmdM}{r}$$
and the total gravitational potential energy is
$$U=\int dU=-Gm\int \dfrac{dM}{r}.$$
We know that $r=\sqrt{x^2+a^2}$ is the same for all segments.
$$U=-\dfrac{Gm}{\sqrt{x^2+a^2}}\int dM=-\dfrac{GmM}{\sqrt{x^2+a^2}}.$$
(b) If $x \gg a$ then $\sqrt{x^2+a^2}=x$ amd $U=-\dfrac{GmM}{x}$.
As expected, this is the gravitational potential energy of two point masses separated by a distance $x$.
(c) Let's use $F_x=-\dfrac{dU}{dx}$
$$F_x=-\dfrac{d}{dx}\left(\dfrac{-GmM}{\sqrt{x^2+a^2}}\right)$$
$$F_x=GmM\dfrac{d}{dx}(x^2+a^2)^{-1/2}=GmM\left(-\dfrac{1}{2}(2x)(x^2+a^2)^{-3/2}\right)=\dfrac{-GmMx}{(x^2+a^2)^{3/2}}$$
The minus sign means the force is attractive.
(d) if $x\gg a$ then $(x^2+a^2)^{3/2}=(x^2)^{3/2}=x^3$
and
$$F_x=-\dfrac{GmMx}{x^3}=-\dfrac{GmM}{x^2}$$
This is the force between two point masses separated by a distance x.
(e) If $x=0$ then $U=-\dfrac{GMm}{a}$ and $F_x=0$. It's obvious, because each small segment of the ring is the same distance from the center and the potential is the same as that due to a point charge of mass $M$ located at a distance $a$. As for force, when the particle is at the center of the ring, symmetrically placed segments of the ring exert equal and opposite forces and the total $F_x$ exerted by the ring is 0.
