Answer
$$\int x^2\sqrt{2x-x^2}dx=\frac{5}{8}\sin^{-1}(x-1)-\frac{x-1}{8}\sqrt{1-(x-1)^2}[1-2(x-1)^2]-\frac{2[1-(x-1)^2]^{3/2}}{3}+\frac{x-1}{2}\sqrt{1-(x-1)^2}+C$$
Work Step by Step
$$A=\int x^2\sqrt{2x-x^2}dx$$ $$A=\int x^2\sqrt{-(x^2-2x)}dx$$ $$A=\int x^2\sqrt{-[(x^2-2x+1)-1]}dx$$ $$A=\int x^2\sqrt{1-(x-1)^2}dx$$
We set $u=x-1$, which means $$du=dx$$
Also, $x^2=(u+1)^2=u^2+2u+1$
Therefore, $$A=\int(u^2+2u+1)\sqrt{1-u^2}du$$ $$A=\int u^2\sqrt{1-u^2}du+2\int u\sqrt{1-u^2}du+\int\sqrt{1-u^2}du$$
The first integral, we use Formula 46, which states that
$$\int x^2\sqrt{a^2-x^2}dx=\frac{a^4}{8}\sin^{-1}\frac{x}{a}-\frac{1}{8}x\sqrt{a^2-x^2}(a^2-2x^2)+C$$
for $a=1$
The second integral, take $a= 1-u^2$, so that $-2udu=da$, therefore, $udu=-\frac{1}{2}da$
The final integral, use Formula 45, which states that
$$\int\sqrt{a^2-x^2}dx=\frac{x}{2}\sqrt{a^2-x^2}+\frac{a^2}{2}\sin^{-1}\frac{x}{a}+C$$
for $a=1$
$$A=\frac{1}{8}\sin^{-1}u-\frac{u}{8}\sqrt{1-u^2}(1-2u^2)+2\int\sqrt{a}(-1/2da)+\frac{u}{2}\sqrt{1-u^2}+\frac{1}{2}\sin^{-1}u+C$$ $$A=\frac{5}{8}\sin^{-1}u-\frac{u}{8}\sqrt{1-u^2}(1-2u^2)-\int a^{1/2}da+\frac{u}{2}\sqrt{1-u^2}+C$$ $$A=\frac{5}{8}\sin^{-1}u-\frac{u}{8}\sqrt{1-u^2}(1-2u^2)-\frac{2a^{3/2}}{3}+\frac{u}{2}\sqrt{1-u^2}+C$$ $$A=\frac{5}{8}\sin^{-1}(x-1)-\frac{x-1}{8}\sqrt{1-(x-1)^2}[1-2(x-1)^2]-\frac{2[1-(x-1)^2]^{3/2}}{3}+\frac{x-1}{2}\sqrt{1-(x-1)^2}+C$$
