Answer
$$\int x\cos^{-1}xdx=\frac{x^2}{2}\cos^{-1}x+\frac{1}{4}\sin^{-1}x-\frac{1}{4}x\sqrt{1-x^2}+C$$
Work Step by Step
$$A=\int x\cos^{-1}xdx$$
Use Formula 107, which states that $$\int x^n\cos^{-1}axdx=\frac{x^{n+1}}{n+1}\cos^{-1}ax+\frac{a}{n+1}\int\frac{x^{n+1}dx}{\sqrt{1-a^2x^2}}$$
for $n=1$ and $a=1$ here:
$$A=\frac{x^2}{2}\cos^{-1}x+\frac{1}{2}\int\frac{x^2dx}{\sqrt{1-x^2}}$$
The remaining integral can be dealt with using Formula 49, which states that
$$\int\frac{x^2}{\sqrt{a^2-x^2}}dx=\frac{a^2}{2}\sin^{-1}\frac{x}{a}-\frac{1}{2}x\sqrt{a^2-x^2}+C$$
with $a=1$:
$$A=\frac{x^2}{2}\cos^{-1}x+\frac{1}{2}\Big(\frac{1}{2}\sin^{-1}x-\frac{1}{2}x\sqrt{1-x^2}\Big)+C$$ $$A=\frac{x^2}{2}\cos^{-1}x+\frac{1}{4}\sin^{-1}x-\frac{1}{4}x\sqrt{1-x^2}+C$$
