Answer
(a) increase
(b) $915rev/s$
Work Step by Step
(a) First of all, we find the speed of light for a complete round trip
$c=\frac{d+d}{\Delta t}$
$c=\frac{2d}{\Delta t}$......eq(1)
Now we find the time taken by an eight sided mirror to complete one round trip
$\Delta t=\frac{\frac{1}{8}rev}{\omega}$
We plug in this value of $\Delta t$ in eq(1) to obtain:
$c=\frac{2d}{\frac{\frac{1}{8}rev}{\omega}}$
$c=16\omega d$
$\implies d=\frac{c}{16\omega}$
The above equation shows that the distance and angular speed are inversely proportional. Thus, if the distance is decreased then the angular speed increases.
(b) We can find the required angular speed as follows:
$\omega=\frac{c}{16d}$
We plug in the known values to obtain:
$\omega=\frac{3\times 10^8m/s}{16(20.5\times 10^3m)}$
$\omega=915rev/s$