Answer
$\dfrac{5}{8}\sin^{-1}(x-1)-\dfrac{x-1}{8}\sqrt{1-(x-1)^2}[1-2(x-1)^2]-\dfrac{2[1-(x-1)^2]^{3/2}}{3}+\dfrac{x-1}{2}\sqrt{1-(x-1)^2}+C$
Work Step by Step
We re-write as $I=\int x^2\sqrt{-[(x^2-2x+1)-1]}dx=\int x^2\sqrt{1-(x-1)^2}dx$
Suppose $u=x-1 \implies du=dx$ and $x^2=(u+1)^2=u^2+2u+1$
Therefore, $I=\int(u^2+2u+1)\sqrt{1-u^2}du\\=\int u^2\sqrt{1-u^2}du+2\int u\sqrt{1-u^2}du+\int\sqrt{1-u^2}du \\=\dfrac{5}{8}\sin^{-1}u-\dfrac{u}{8}\sqrt{1-u^2}(1-2u^2)-\int u^{1/2}du+\dfrac{u}{2}\sqrt{1-u^2}+C \\=\dfrac{5}{8}\sin^{-1}u-\dfrac{u}{8}\sqrt{1-u^2}(1-2u^2)-\dfrac{2u^{3/2}}{3}+\dfrac{u}{2}\sqrt{1-u^2}+C\\=\dfrac{5}{8}\sin^{-1}(x-1)-\dfrac{x-1}{8}\sqrt{1-(x-1)^2}[1-2(x-1)^2]-\dfrac{2[1-(x-1)^2]^{3/2}}{3}+\dfrac{x-1}{2}\sqrt{1-(x-1)^2}+C$
