Answer
$ 6.1\times10^{-7}\;\rm N$ upward.
Work Step by Step
From the geometry of the first figure below, we can see that the two masses B and C are at the same distance from mass A. Also, we can see that the right triangle, shaped by A, B, and $y$-axis, is identical to the right triangle shaped by A, C, and $y$-axis.
This means that the net force exerted on mass A is directly upward, exactly on the $y$-axis while the two $x$-force components are canceling each other.
Thus, the net force exerted on mass A is given by
$$F=\sum F_y=F_{\rm B\;on A}\cos\theta+F_{\rm C\;on A}\cos\theta$$
$$F=\cos\theta\left[\dfrac{Gm_Am_B}{r_{AB}^2}+\dfrac{Gm_Am_C}{r_{AC}^2}\right]$$
$$F=Gm_A\cos\theta\left[\dfrac{m_B}{r_{AB}^2}+\dfrac{ m_C}{r_{AC}^2}\right]$$
Plugging the known from the figure below;
$$F=(6.67\times10^{-11})(20)\cdot \dfrac{20}{5\sqrt{17}}\left[\dfrac{10}{(5\sqrt{17}\times10^{-2})^2}+\dfrac{10}{(5\sqrt{17}\times10^{-2})^2}\right]$$
$$F= \color{red}{\bf 6.1\times10^{-7}}\ \;\rm N\tag {Upward}$$
