Answer
See a detailed answer below.
Work Step by Step
First, we need to write the mentioned equations.
$\bullet$ Equation 14.54:
$$\dfrac{d^2x}{dt^2}+\dfrac{b}{m}\dfrac{dx}{dt}+\dfrac{k}{m}x=0\tag 1$$
$\bullet$ Equation 14.55:
$$x_{(t)} = Ae^{-bt/2m} \cos(\omega t + \phi_0)\tag 2$$
$\bullet$ Equation 14.56:
$$\omega=\sqrt{\dfrac{k}{m}- \dfrac{b^2}{4m^2}}\tag 3$$
The author asked us to prove that (2) is a solution of (1) if, and only if, the angular frequency is given by (3).
Taking the first derivative of $x_{(t)}$ in (2):
$$\dfrac{dx}{dt} =\dfrac{d}{dt}\left[ Ae^{-bt/2m} \cos(\omega t + \phi_0)\right]$$
$$\dfrac{dx}{dt} =\dfrac{ -bA}{2m}e^{-bt/2m} \cos(\omega t + \phi_0) -A\omega e^{-bt/2m} \sin(\omega t + \phi_0) $$
$$\dfrac{dx}{dt} =e^{-bt/2m}\left(\dfrac{ -bA}{2m} \cos(\omega t + \phi_0) -A\omega \sin(\omega t + \phi_0) \right)\tag 4$$
Taking the second derivative;
$$\dfrac{d^2x}{dt^2} =\dfrac{d}{dt}\left[e^{-bt/2m}\left(\dfrac{ -bA}{2m} \cos(\omega t + \phi_0) -A\omega \sin(\omega t + \phi_0) \right)\right]$$
$$\dfrac{d^2x}{dt^2} =\dfrac{-b}{2m} e^{-bt/2m}\left(\dfrac{ -bA}{2m} \cos(\omega t + \phi_0) -A\omega \sin(\omega t + \phi_0) \right) +\\
e^{-bt/2m}\left(\dfrac{ -(-\omega)bA}{2m} \sin(\omega t + \phi_0) -A\omega^2 \cos(\omega t + \phi_0) \right) $$
$$\dfrac{d^2x}{dt^2} =\dfrac{b^2A}{4m^2} e^{-bt/2m} \cos(\omega t + \phi_0) +\dfrac{ bA\omega }{2m} e^{-bt/2m} \sin(\omega t + \phi_0) +\\
\dfrac{ bA\omega}{2m} e^{-bt/2m} \sin(\omega t + \phi_0) - A\omega^2 e^{-bt/2m} \cos(\omega t + \phi_0) $$
$$\dfrac{d^2x}{dt^2} =e^{-bt/2m}\left[\dfrac{b^2A}{4m^2} \cos(\omega t + \phi_0) - A\omega^2 \cos(\omega t + \phi_0) +\\
\dfrac{ bA\omega }{2m} \sin(\omega t + \phi_0) +
\dfrac{ bA\omega}{2m} \sin(\omega t + \phi_0) \right]$$
$$\dfrac{d^2x}{dt^2} =e^{-bt/2m}\left[\cos(\omega t + \phi_0)\left(\dfrac{b^2A}{4m^2} - A\omega^2 \right)+ \\
\sin(\omega t + \phi_0) \left(\dfrac{ bA\omega }{2m} +
\dfrac{ bA\omega}{2m} \right) \right]$$
$$\dfrac{d^2x}{dt^2} =e^{-bt/2m}\left[\cos(\omega t + \phi_0)\left(\dfrac{b^2A}{4m^2} - A\omega^2 \right)+ \\
\sin(\omega t + \phi_0) \left(\dfrac{ bA\omega }{ m} \right) \right]\tag 5$$
Plugging (4) and (5) into (1);
$$ e^{-bt/2m}\left[\cos(\omega t + \phi_0)\left(\dfrac{b^2A}{4m^2} - A\omega^2 \right)+ \\
\sin(\omega t + \phi_0) \left(\dfrac{ bA\omega }{ m} \right) \right]+\\
\dfrac{b}{m}e^{-bt/2m}\left(\dfrac{ -bA}{2m} \cos(\omega t + \phi_0) -A\omega \sin(\omega t + \phi_0) \right)+\\
\dfrac{k}{m}x=0 $$
$$ e^{-bt/2m} \left[\cos(\omega t + \phi_0)\left(\dfrac{b^2A}{4m^2} - A\omega^2 \right)+
\sin(\omega t + \phi_0) \left(\dfrac{ bA\omega }{ m} \right) \\
+\dfrac{ -b^2A}{2m^2} \cos(\omega t + \phi_0) - \dfrac{bA\omega}{m} \sin(\omega t + \phi_0) \right] +\\
\dfrac{k}{m}x=0 $$
The term of $ \dfrac{bA\omega}{m} \sin(\omega t + \phi_0) $ cancel.
$$ e^{-bt/2m}\cos(\omega t + \phi_0) \left[ \dfrac{b^2A}{4m^2} - A\omega^2
+\dfrac{ -b^2A}{2m^2} \right] +
\dfrac{k}{m}x=0 $$
Plugging $x$ from (2);
$$ e^{-bt/2m}\cos(\omega t + \phi_0) \left[ \dfrac{b^2A}{4m^2} - A\omega^2 +\dfrac{ -b^2A}{2m^2} \right] + \dfrac{k}{m} Ae^{-bt/2m} \cos(\omega t + \phi_0)=0 $$
$$A e^{-bt/2m}\cos(\omega t + \phi_0) \left[ \dfrac{b^2 }{4m^2} - \omega^2 +\dfrac{ -b^2 }{2m^2} + \dfrac{k }{m} \right] =0 $$
Dividing both sides by $A e^{-bt/2m}\cos(\omega t + \phi_0)$, so
$$ \dfrac{b^2 }{4m^2} - \omega^2 +\dfrac{ -b^2 }{2m^2} + \dfrac{k }{m} =0 $$
Solving for $\omega^2$;
$$ \omega^2 = \dfrac{b^2 -2b^2 }{4m^2} + \dfrac{k }{m} = \dfrac{-b^2 }{4m^2} + \dfrac{k }{m} $$
Therefore,
$$ \omega =\sqrt{\dfrac{k }{m} - \dfrac{ b^2 }{4m^2} } $$
which is equation (3) as the author asked.