Answer
$$\int x^2\tan^{-1}xdx=\frac{x^3}{3}\tan^{-1}x-\frac{1}{3}\Big(\frac{x^2}{2}-\frac{1}{2}\ln(1+x^2)\Big)+C$$
Work Step by Step
$$A=\int x^2\tan^{-1}xdx$$
Use Formula 108, which states that $$\int x^n\tan^{-1}axdx=\frac{x^{n+1}}{n+1}\tan^{-1}ax-\frac{a}{n+1}\int\frac{x^{n+1}dx}{1+a^2x^2}$$
for $n=2$ and $a=1$ here:
$$A=\frac{x^3}{3}\tan^{-1}x-\frac{1}{3}\int\frac{x^3dx}{1+x^2}$$ $$A=\frac{x^3}{3}\tan^{-1}x-\frac{1}{3}\Big(\int\frac{x^3+x}{1+x^2}dx-\int\frac{x}{1+x^2}dx\Big)$$ $$A=\frac{x^3}{3}\tan^{-1}x-\frac{1}{3}\Big(\int xdx-\frac{1}{2}\int\frac{2xdx}{1+x^2}\Big)$$ $$A=\frac{x^3}{3}\tan^{-1}x-\frac{1}{3}\Big(\frac{x^2}{2}-\frac{1}{2}\int\frac{1}{1+x^2}d(1+x^2)\Big)$$ $$A=\frac{x^3}{3}\tan^{-1}x-\frac{1}{3}\Big(\frac{x^2}{2}-\frac{1}{2}\ln|1+x^2|\Big)+C$$
Since $1+x^2\gt0$ for all $x$, $|1+x^2|=1+x^2$
$$A=\frac{x^3}{3}\tan^{-1}x-\frac{1}{3}\Big(\frac{x^2}{2}-\frac{1}{2}\ln(1+x^2)\Big)+C$$
