Chapter 7 - Analytic Trigonometry - 7.2 The Inverse Trigonometric Functions (Continued) - 7.2 Assess Your Understanding - Page 458: 21

$\dfrac{3\pi}{4}$

Note that $\sin{\left(\dfrac{5\pi}{4}\right)}=\dfrac{\sqrt2}{2}$. Thus $\cos^{-1}\left(\sin{\left(\dfrac{5\pi}{4}\right)}\right)=\cos^{-1}{\left(\dfrac{\sqrt2}{2}\right)}=\dfrac{3\pi}{4}$, because $\cos{\left(\dfrac{3\pi}{4}\right)}=\dfrac{\sqrt2}{2}$ and $\dfrac{3\pi}{4}$ is in the range of $\cos^{-1}{x}$, which is $\left[0,\pi\right].$