Answer
$-1.82\times10^{-7}\;\rm J$
Work Step by Step
We know that the total gravitational potential energy of a system is the sum of the potential energies.
$$U_{sys}=-U_{AB}-U_{AC}-U_{BC}$$
$$U_{sys}=-\left[U_{AB}+U_{AC}+U_{BC}\right]$$
$$U_{sys}=-\left[\dfrac{Gm_Am_B}{r_{AB}}+\dfrac{Gm_Am_C}{r_{AC}}+\dfrac{Gm_Bm_C}{r_{BC}}\right]$$
$$U_{sys}=-G\left[\dfrac{ m_Am_B}{r_{AB}}+\dfrac{ m_Am_C}{r_{AC}}+\dfrac{ m_Bm_C}{r_{BC}}\right]$$
Plugging the now from the geometry of the figure below;
$$U_{sys}=-6.67\times10^{-11}\left[\dfrac{ (20)(10)}{0.10}+\dfrac{ (20)(5)}{0.20}+\dfrac{(10)(5)}{(10\sqrt 5)\times10^{-2}}\right]$$
$$U_{sys}=\color{red}{\bf -1.82\times10^{-7}}\;\rm J$$
