Answer
a) $U_{total}=-1.46\times 10^{-8}J$
b) $\times4$
c) $\times2$
Work Step by Step
(a) We know that
$U_{total}=-G[\frac{m_1m_2}{AB}+\frac{m_2m_3}{BC}+\frac{m_3m_4}{CD}+\frac{m_4m_1}{DA}+\frac{m_1m_3}{AC}+\frac{m_2m_4}{DB}]$
We plug in the known values to obtain:
$U_{total}=-(6.67\times 10^{-11}Nm^2/Kg^2)[(\frac{2.0Kg^2}{0.20m})+(\frac{6.0Kg^2}{0.10m})+(\frac{12.0Kg^2}{0.20m})+(\frac{4.0Kg^2}{0.10m})+(\frac{3.0Kg^2}{0.224m})+(\frac{8.0Kg^2}{0.224m})]$
This simplifies to:
$U_{total}=-1.46\times 10^{-8}J$
(b) We know that if the masses in the system are doubled then the gravitational potential energy of the system becomes four times as great (quadruples).
(c) We know that if all the sides of the rectangle are halved, then the gravitational potential energy of the system becomes double what it used to be.