Chapter 7 - Analytic Trigonometry - 7.6 Double-angle and Half-angle Formulas - 7.6 Assess Your Understanding - Page 498: 89

$\frac{1}{5}$

Step 1. Letting $cos^{-1}(\frac{3}{5})=t$, we have $cos(t)=\frac{3}{5}$ and $t\in (0,\frac{\pi}{2})$ Step 2. Thus $sin^2(\frac{1}{2}cos^{-1}(\frac{3}{5}))=sin^2(\frac{1}{2}t)=\frac{1-cos(t)}{2}=\frac{1-\frac{3}{5}}{2}=\frac{1}{5}$