Answer
The mass of particle B is $~~0.50~kg$
The mass of particle C is $~~0.50~kg$
Work Step by Step
As $y \to \infty$, particle A contributes zero to the gravitational potential energy of the system.
We can assume that the gravitational potential energy of the two-particle system consisting of B and C is $-2.7\times 10^{-11}~J$
We can find the masses of particles B and C:
$U = -\frac{GM^2}{2D} = -2.7\times 10^{-11}~J$
$M^2 = \frac{(2.7\times 10^{-11}~J)(2D)}{G}$
$M = \sqrt{\frac{(2.7\times 10^{-11}~J)(2D)}{G}}$
$M = \sqrt{\frac{(2.7\times 10^{-11}~J)(2)(0.3057~m)}{6.67\times 10^{-11}~N~m^2/kg^2}}$
$M = 0.50~kg$
The mass of particle B is $~~0.50~kg$
The mass of particle C is $~~0.50~kg$