Answer
$$\int_{\frac{\pi}{4}}^{\frac{\pi}{2}} cot^{5}x\,csc^{3}x\,dx=\frac{22\sqrt{2}-8}{105} $$
Work Step by Step
$$\int cot^{5}x\,csc^{3}x\,dx=\int(csc^{2}x-1)^{2}\,csc^{2}x\,csc\,x\,cot\,x\,dx$$
$$Let\ t=csc\,x,dt=-csc\,x\,cot\,x\,dx$$
$$\int cot^{5}x\,csc^{3}x\,dx=-\int (t^{2}-1)^{2}t^{2}dt$$
$$=-\int (t^{6}-2t^{4}+t^{2})dt$$
$$=-\frac{1}{7}t^{7}+\frac{2}{5}t^{5}-\frac{1}{3}t^{3}+C$$
$$=-\frac{1}{7}csc^{7}x+\frac{2}{5}csc^{5}x-\frac{1}{3}csc^{3}x+C$$
$$\int_{\frac{\pi}{4}}^{\frac{\pi}{2}} cot^{5}x\,csc^{3}x\,dx=\left [ -\frac{1}{7}csc^{7}x+\frac{2}{5}csc^{5}x-\frac{1}{3}csc^{3}x \right ]_{\frac{\pi}{4}}^{\frac{\pi}{2}}$$
$$=\left [ -\frac{8}{105}-\left (-\frac{22\sqrt{2}}{105} \right ) \right ]=\frac{22\sqrt{2}-8}{105} $$