Answer
See explanation.
Work Step by Step
We calculate the total number of states with principal quantum number n. Add up all the possibilities.
The maximum $l$ value is (n-1), and each $l$ value has $2l+1$ different $m_l$ values.
Also, the spin states give us a factor of 2.
The total number of states is:
$$N= 2\sum_{l=0}^{n-1}2l+1 = 2\sum_{l=0}^{n-1}2l +2\sum_{l=0}^{n-1}1 $$
$$N= 4\sum_{l=0}^{n-1}l +4\sum_{l=0}^{n-1}1 $$
Use the hint given in the problem.
$$N= 4\frac{(n-1)n}{2} +2n=\frac{4n^2-4n}{2} +2n $$
$$N= 2n^2-2n +2n =2n^2$$
This was to be shown.
b. The n=5 shell (O-shell) has $2(5)^2=50$ states.
