Answer
$$\frac{x^{3}}{3} \tan ^{-1} x-\frac{x^{2}}{6}+\frac{1}{6} \ln \left(1+x^{2}\right)+C$$
Work Step by Step
Use the formula $$\int x^{n} \tan ^{-1} a x d x=\frac{x^{n+1}}{n+1} \tan ^{-1} a x-\frac{a}{n+1} \int \frac{x^{n+1} d x}{1+a^{2} x^{2}}, \quad n \neq-1$$
with $n=2,\ a=1$
\begin{align*}
\int x^{2} \tan ^{-1} x d x&=\frac{x^{2+1}}{2+1} \tan ^{-1} x-\frac{1}{2+1} \int \frac{x^{2+1}}{1+x^{2}} d x\\
&=\frac{x^{3}}{3} \tan ^{-1} x-\frac{1}{3} \int \frac{x^{3}}{1+x^{2}} d x\\
&=\frac{x^{3}}{3} \tan ^{-1} x-\frac{1}{3} \left(\int x d x-\int \frac{x d x}{1+x^{2}}\right) \\
&=\frac{x^{3}}{3} \tan ^{-1} x-\frac{1}{3} \left(\frac{x^{2}}{2}-\frac{1}{2} \ln \left(1+x^{2}\right)\right)+C\\
&=\frac{x^{3}}{3} \tan ^{-1} x-\frac{x^{2}}{6}+\frac{1}{6} \ln \left(1+x^{2}\right)+C
\end{align*}
