Answer
$$\int x^2\tan^{-1}\frac{x}{2}dx=\frac{1}{3}\Big(x^3\tan^{-1}\frac{x}{2}-x^2+4\ln(4+x^2)\Big)+C$$
Or:
$$\frac{1}{3}\Big(x^3\tan^{-1}\frac{x}{2}-x^2+4\ln(1+\frac{x^2}{4})\Big)+C$$
Work Step by Step
$$A=\int x^2\tan^{-1}\frac{x}{2}dx$$
We set $u=\tan^{-1}\frac{x}{2}$ and $dv=x^2dx$
This means $$du=\frac{(\frac{x}{2})'}{1+(\frac{x}{2})^2}dx=\frac{\frac{1}{2}}{1+\frac{x^2}{4}}dx=\frac{1}{2(1+\frac{x^2}{4})}dx$$ and $$v=\frac{x^3}{3}$$
Applying the formula $\int udv=uv-\int vdu$, we have $$A=\frac{x^3\tan^{-1}\frac{x}{2}}{3}-\frac{1}{6}\int\frac{x^3}{1+\frac{x^2}{4}}dx$$ $$A=\frac{x^3\tan^{-1}\frac{x}{2}}{3}-\frac{1}{6}\int\frac{x^3}{\frac{4+x^2}{4}}dx$$ $$A=\frac{x^3\tan^{-1}\frac{x}{2}}{3}-\frac{1}{6}\int\frac{4x^3}{4+x^2}dx$$ $$A=\frac{x^3\tan^{-1}\frac{x}{2}}{3}-\frac{2}{3}\int\frac{x^2}{4+x^2}(xdx)$$
We take $a=4+x^2$, or $x^2=a-4$ That means $$da=2xdx$$ $$xdx=\frac{1}{2}da$$ Therefore, $$A=\frac{x^3\tan^{-1}\frac{x}{2}}{3}-\frac{2}{3}\int\frac{a-4}{a}\Big(\frac{1}{2}da\Big)$$ $$A=\frac{x^3\tan^{-1}\frac{x}{2}}{3}-\frac{1}{3}\int\Big(1-\frac{4}{a}\Big)da$$ $$A=\frac{x^3\tan^{-1}\frac{x}{2}}{3}-\frac{1}{3}\Big(\int1da-\int\frac{4}{a}da\Big)$$ $$A=\frac{x^3\tan^{-1}\frac{x}{2}}{3}-\frac{1}{3}\Big(a-4\ln |a|\Big)+C$$ $$A=\frac{x^3\tan^{-1}\frac{x}{2}}{3}-\frac{4+x^2-4\ln|4+x^2|}{3}+C$$
However, since $x^2+4\gt0$ for all $x$, we always have $|4+x^2|=4+x^2$ $$A=\frac{x^3\tan^{-1}\frac{x}{2}}{3}-\frac{4+x^2-4\ln(4+x^2)}{3}+C$$ $$A=\frac{1}{3}\Big(x^3\tan^{-1}\frac{x}{2}-4-x^2+4\ln(4+x^2)\Big)+C$$
$$A=\frac{1}{3}\Big(x^3\tan^{-1}\frac{x}{2}-x^2+4\ln(4+x^2)\Big)+C$$
We can remove arbitrary constants. An alternative form is possible:
$$A=\frac{1}{3}\Big(x^3\tan^{-1}\frac{x}{2}-x^2+4\ln(1+\frac{x^2}{4})\Big)+C$$
