Answer
$$\int\cot^62xdx=-\frac{\cot^52x}{10}+\frac{\cot^32x}{6}-\frac{\cot2x}{2}-x+C$$
Work Step by Step
$$A=\int\cot^62xdx=\frac{1}{2}\int\cot^62xd(2x)$$ $$A=\frac{1}{2}\int\cot^6ada$$ (set $2x=a$)
$$A=\frac{1}{2}\int\cot^2a(\cot^2a)^2da$$ $$A=\frac{1}{2}\int\cot^2a(\csc^2a-1)^2da$$ $$A=\frac{1}{2}\int\cot^2a(\csc^4a-2\csc^2a+1)da$$ $$A=\frac{1}{2}\int\cot^2a\csc^4ada-\int\cot^2a\csc^2ada+\frac{1}{2}\int\cot^2ada$$ $$A=X-Y+Z$$
1) For $X$: $$X=\frac{1}{2}\int\cot^2a\csc^4ada=\frac{1}{2}\int\cot^2a\csc^2a(\csc^2ada)$$
Set $u=\cot a$, which means $du=-\csc^2ada$, so $\csc^2ada=-du$
We also have $\csc^2a=\cot^2a+1=u^2+1$
Therefore, $$X=-\frac{1}{2}\int u^2(u^2+1)du=-\frac{1}{2}\int(u^4+u^2)du$$ $$X=-\frac{1}{2}\Big(\frac{u^5}{5}+\frac{u^3}{3}\Big)+C=-\frac{u^5}{10}-\frac{u^3}{6}+C$$ $$X=-\frac{\cot^52x}{10}-\frac{\cot^32x}{6}+C$$
2) For $Y$: $$Y=\int\cot^2a\csc^2ada=-\int\cot^2a d(\cot a)$$ $$Y=-\frac{\cot^3a}{3}+C=-\frac{\cot^32x}{3}+C$$
3) For $Z$: $$Z=\frac{1}{2}\int\cot^2ada=\frac{1}{2}\int(\csc^2a-1)da$$ $$Z=\frac{1}{2}(-\cot a-a)+C=-\frac{a+\cot a}{2}+C$$ $$Z=-\frac{2x+\cot2x}{2}+C$$
Therefore, $$A=-\frac{\cot^52x}{10}-\frac{\cot^32x}{6}+\frac{\cot^32x}{3}-\frac{2x+\cot2x}{2}+C$$ $$A=-\frac{\cot^52x}{10}+\frac{\cot^32x}{6}-\frac{\cot2x}{2}-x+C$$
