Answer
a) $3.33\;\rm m/s$
b) $11.7\;\rm cm$
c) $0.833\;\rm m/s$, $6.45\;\rm cm$
Work Step by Step
We chose the system to be the ball plus the block plus the spring, and we assume that this system is an isolated system in the $x$-direction since the table is frictionless. We assume that the spring is an ideal one with a negligible mass.
a) The momentum is conserved since the system is an isolated one.
And since the collision is perfectly elastic, so the final speed of the ball just after the collision is given by
$$v_{fx,ball}=\dfrac{(m_{ball}-m_{block})v_{ix,ball}}{m_{ball}+m_{block}}$$
We chose the system here to be the ball plus the block.
Plugging the given;
$$v_{fx,ball}=\dfrac{(0.02-0.1)\times 5.0}{0.02+0.1}={\bf -3.33}\;\rm m/s$$
Thus, the ball's speed is
$$v_{fx,ball}=\color{red}{\bf 3.33}\;\rm m/s$$
___________________________________________
b) To find the compressed distance by the spring, we need to find the velocity of the block just before the collision and then we can use the conservation of energy law to find the compressed distance by the spring.
$$v_{fx,block}=\dfrac{ 2m_{ball} v_{ix,ball}}{m_{ball}+m_{block}}$$
Plugging the given;
$$v_{fx,ball}=\dfrac{ 2\times 0.02 \times 5.0}{0.02+0.1}={\bf 1.667}\;\rm m/s$$
The energy is conserved, so
$$E_i=E_f$$
$$K_{i,ball}+K_{i,block}+U_{is}=K_{f,ball}+K_{f,block}+U_{fs}$$
$$\color{red}{\bf\not} \frac{1}{2}m_{ball}v_{ix,ball}^2+0+0=\color{red}{\bf\not} \frac{1}{2}m_{ball}v_{fx,ball}^2+\color{red}{\bf\not} \frac{1}{2}m_{block}v_{fx,block}^2+\color{red}{\bf\not} \frac{1}{2}kx^2$$
The final speed of the block is zero as well since the spring is compressed and stopped.
$$ m_{ball}v_{ix,ball}^2 = m_{ball}v_{fx,ball}^2+ 0+ kx^2$$
Solving for $x$; which is the compressed distance by the spring.
$$ m_{ball}v_{ix,ball}^2- m_{ball}v_{fx,ball}^2 = kx^2$$
$$ x =\sqrt{\dfrac{ m_{ball}v_{ix,ball}^2- m_{ball}v_{fx,ball}^2 }{k}} $$
Plugging the known;
$$ x =\sqrt{\dfrac{ [0.02\times 5^2]- [0.02\times (-3.33)^2] }{20}} =\bf 0.1179 \;\rm m$$
Thus,
$$x=\color{red}{\bf 11.8}\;\rm cm$$
___________________________________________
c) If the collision is perfectly inelastic, so the block and the ball stick together after the collision.
We chose the system here to be the ball plus the block.
Thus,
$$p_{ix}=p_{fx}$$
$$m_{ball}v_{ix,ball}=(m_{ball}+m_{block})v_{fx}$$
Hence,
$$v_{fx}=\dfrac{0.02\times 5}{0.02+0.1}=\color{red}{\bf 0.833}\;\rm m/s$$
The same approach gives the compressed distance we did above;
$$E_i=E_f$$
$$K_{ix,(ball+block)}+U_{is}=K_{fx,(ball+block)}+U_{fs}$$
$$\color{red}{\bf\not} \frac{1}{2}(m_{ball}+m_{block})v_{ix}^2+0=0+\color{red}{\bf\not} \frac{1}{2}kx^2$$
$$(m_{ball}+m_{block})v_{ix}^2 = 0+ 0+ kx^2$$
where $v_{ix}=v_{fx}$ here is the speed just after the collision.
Solving for $x$; which is the compressed distance by the spring.
$$ x =\sqrt{\dfrac{ (m_{ball}+m_{block})v_{ix}^2 }{k}} $$
Plugging the known;
$$ x =\sqrt{\dfrac{ (0.02+0.1) \times 0.833^2 }{20}} =\bf 0.0645\;\rm m$$
Thus,
$$x=\color{red}{\bf 6.45}\;\rm cm$$
