Answer
See the detailed answer below.
Work Step by Step
We are given that the distance between the centers of the two spheres initially was $10R$ and we need to find their velocities just before they collide.
Noting that the distance between their centers before the collision is the sum of their radius and is then $2R$.
We can assume that the two spheres are isolated which means that the energy and the momentum of the system are conserved.
$$E_i=E_f$$
$$K_i+U_{gi}=K_f+U_{gf}$$
The two spheres start from rest, so $K_i=0$
$$0+\dfrac{-Gm_1m_2}{r_i}=\frac{1}{2}m_1v_{1f}^2+\frac{1}{2}m_2v_{2f}^2+\dfrac{-Gm_1m_2}{r_f}$$
Noting that $m_1=M$, $m_2=2M$, $r_i=10R$, and $r_f=2R$
$$ \dfrac{-2GM^{\color{red}{\bf\not}2} }{ 10R }=\frac{1}{2}\color{red}{\bf\not}Mv_{1f}^2+\color{red}{\bf\not}Mv_{2f}^2+\dfrac{-2GM^{\color{red}{\bf\not}2}}{2R}$$
$$ \dfrac{-2GM }{ 10R }=\frac{1}{2} v_{1f}^2+ v_{2f}^2+\dfrac{-2GM }{2R}$$
Hence,
$$ \frac{1}{2} v_{1f}^2+ v_{2f}^2=\dfrac{-2GM }{ 10R }+\dfrac{2GM }{2R}=\dfrac{-2GM +10GM}{10R}$$
$$ \frac{1}{2} v_{1f}^2+ v_{2f}^2= \dfrac{4GM}{5R}\tag1$$
The conservation of momentum;
$$p_i=p_f$$
The two spheres start from rest, so $p_i=0$
$$0=p_f$$
$$0=m_1v_{1f}-m_2v_{2f}$$
The negative sign is due to the velocity direction, as seen in the figure below, where we chose right to be the positive direction which is the direction of the sphere's 1 velocity.
Hence,
$$ \color{red}{\bf\not}Mv_{1f}=2 \color{red}{\bf\not}Mv_{2f}$$
$$v_{1f}=2v_{2f}\tag 2$$
Plugging into (1);
$$ \frac{1}{2} (2v_{2f})^2+ v_{2f}^2= \dfrac{4GM}{5R} $$
$$ 2v_{2f}^2+ v_{2f}^2= \dfrac{4GM}{5R} $$
$$3 v_{2f}^2= \dfrac{4GM}{5R} $$
$$ v_{2f} = \sqrt{\dfrac{4GM}{15R} }$$
$$ \boxed{v_{2f} =2\left( \sqrt{\dfrac{ GM}{15R} }\right)}$$
Plugging into (2);
$$ \boxed{v_{1f} =4\left( \sqrt{\dfrac{ GM}{15R} }\right)}$$