Chapter 6 - Analytic Trigonometry - Section 6.2 The Inverse Trigonometric Functions (Continued) - 6.2 Assess Your Understanding - Page 480: 21

$\dfrac{3 \pi}{4}$

The range of $\cos^{-1}{x}$ is $[0, \pi]$. Here, $\sin(\dfrac{ 5\pi}{4})=\dfrac{\sqrt 2}{2}$, because $\cos{(\dfrac{3\pi}{4})}=\dfrac{\sqrt 2}{2}$ and $\dfrac{ 3\pi}{4}$ lies in the range of $\cos^{-1}{x}$ . Therefore, we have: $\cos^{-1}(\sin {(\dfrac{5 \pi }{4})}]=\cos^{-1} ({\dfrac{ \sqrt 2}{2}})=\dfrac{3 \pi}{4}$