Answer
a) $1.334\times10^{-6}\;\rm N$, $\approx 83^\circ$
a) $2.28\times10^{-7}\;\rm N$, $\approx 7.5^\circ$
Work Step by Step
a) We need to draw the force diagram exerted on the 20-kg mass (A).
We can find the force exerted on A in both directions by applying Newton's gravitational law as follows;
$$ F_x=F_{\rm B\;on\;A}=\dfrac{Gm_Am_B}{r_{AB}^2}$$
$$F_x=\dfrac{(6.672\times10^{-11})(20)(10)}{(0.1)^2}=\bf 1.334\times10^{-6}\;\rm N$$
$$ F_y=F_{\rm C\;on\;A}=\dfrac{Gm_Am_C}{r_{AC}^2}$$
$$F_y=\dfrac{(6.672\times10^{-11})(20)(5)}{(0.2)^2}=\bf 1.668\times10^{-7}\;\rm N$$
Now we can use the Pythagorean theorem to find the net force exerted on A.
$$F=\sqrt{F_x^2+F_y^2}$$
Plugging from above;
$$F=\sqrt{(1.334\times10^{-6})^2+(1.668\times10^{-7})^2}$$
$$\sum F=\color{red}{\bf1.344\times10^{-6}}\;\rm N$$
And its direction relative to $y-$direction is given by
$$\theta =\tan^{-1}\left[\dfrac{F_x}{F_y}\right]$$
Plugging from above;
$$\theta =\tan^{-1}\left[\dfrac{1.334\times10^{-6} }{1.668\times10^{-7} }\right]$$
$$\theta=\color{red}{\bf82.9}^\circ\tag{clockwise from $y$-axis}$$
----
By the same approach, and from the geometry of the figure below;
$$ F_x=F_{\rm B\;on\;C}\sin\theta=\dfrac{Gm_Cm_B}{r_{CB}^2}\sin\theta$$
$$F_x=\dfrac{(6.672\times10^{-11})(5)(10)}{(10\sqrt5
\times10^{-2})^2}\cdot \dfrac{10}{10\sqrt5}=\bf 2.984\times10^{-8}\;\rm N$$
$$ F_y=-F_{\rm A\;on\;C}-F_{\rm B\;on\;C}\cos\theta=-\dfrac{Gm_Am_C}{r_{AC}^2}-\dfrac{Gm_Cm_B}{r_{CB}^2}\cos\theta$$
$$F_y=-\dfrac{(6.672\times10^{-11})(20)(5)}{(0.2)^2}-\dfrac{(6.672\times10^{-11})(5)(10)}{(10\sqrt5
\times10^{-2})^2}\cdot \dfrac{20}{10\sqrt5}$$
$$F_y=\bf -2.2648\times10^{-7}\;\rm N$$
Now we can use the Pythagorean theorem to find the net force exerted on C.
$$F=\sqrt{F_x^2+F_y^2}$$
Plugging from above;
$$F=\sqrt{(2.984\times10^{-8} )^2+(-2.2648\times10^{-7})^2}$$
$$\sum F=\color{red}{\bf 2.28\times10^{-7}}\;\rm N$$
And its direction relative to $y-$direction is given by
$$\theta =\tan^{-1}\left[\dfrac{F_x}{F_y}\right]$$
Plugging from above;
$$\theta =\tan^{-1}\left[\dfrac{2.984\times10^{-8} }{2.2648\times10^{-7} }\right]$$
$$\theta=\color{red}{\bf 7.5}^\circ\tag{counterclockwise from $y$-axis}$$
