Answer
The total gravitational potential energy of the three masses is $-1.96\times 10^{-7}~J$
Work Step by Step
We can find the distance $r_1$.
$r_1 = \sqrt{(5.0~cm)^2+(20.0~cm)^2}$
$r_1 = 20.6~cm$
Note that $r_3$ is also 20.6 cm
We can find the gravitational potential energy of $m_1$ and $m_2$.
$U_{1,2} = -\frac{G~m_1~m_2}{r_1}$
$U_{1,2} = -\frac{(6.67\times 10^{-11}~m^3/kg~s^2)(20.0~kg)(10.0~kg)}{(0.206~m)}$
$U_{1,2} = -6.48\times 10^{-8}~J$
We can find the gravitational potential energy of $m_2$ and $m_3$.
$U_{2,3} = -\frac{G~m_2~m_3}{r_2}$
$U_{2,3} = -\frac{(6.67\times 10^{-11}~m^3/kg~s^2)(10.0~kg)(10.0~kg)}{(0.10~m)}$
$U_{2,3} = -6.67\times 10^{-8}~J$
We can find the gravitational potential energy of $m_1$ and $m_3$.
$U_{1,3} = -\frac{G~m_1~m_3}{r_3}$
$U_{1,3} = -\frac{(6.67\times 10^{-11}~m^3/kg~s^2)(20.0~kg)(10.0~kg)}{(0.206~m)}$
$U_{1,3} = -6.48\times 10^{-8}~J$
To find the total gravitational potential energy, we can add the gravitational potential energy of each pair of masses.
$U_{total} = U_{1,2}+U_{2,3}+U_{1,3}$
$U_{total} = (-6.48\times 10^{-8}~J)+(-6.67\times 10^{-8}~J)+(-6.48\times 10^{-8}~J)$
$U_{total} = -1.96\times 10^{-7}~J$
The total gravitational potential energy of the three masses is $-1.96\times 10^{-7}~J$