Answer
$$\frac{1}{2}x\sqrt{x^2+1}-\frac{1}{2}\ln \left|x+\sqrt{x^2+1}\right|+C$$
Work Step by Step
Given
$$\int \frac{x^2}{\sqrt{x^2+1}} d x$$
Apply Trigonometric Substitution
\begin{aligned}
x&=\tan u\ \ \ \to \ \ dx=\sec^2 udu
\end{aligned}
Then
\begin{aligned}
\int \frac{x^2}{\sqrt{x^2+1}} d x&= \int \frac{\tan^2u }{\sqrt{\tan^2u+1}} \sec^2u du\\
&= \int \tan^2 u \sec udu \\
&=\int \left(-1+\sec ^2\left(u\right)\right)\sec \left(u\right)du\\
&=-\int \sec \left(u\right)du+\int \sec ^3\left(u\right)du
\end{aligned}
Now, evaluate
\begin{aligned}
\int \sec ^3\left(u\right)du\end{aligned}
Let
\begin{aligned} U&= \sec u \ \ \ \ \ \ dV= \sec^2udu\\
dU&= \sec u\tan u du \ \ \ \ \ \ V = \tan u\end{aligned}
Then
\begin{aligned}\int \sec ^3\left(u\right)du&= \sec u\tan u -\int \sec u\tan^2 udu\\
&= \sec u\tan u -\int \sec u(\sec^2 u-1)du\\
2\int \sec^3 u du&= \sec u\tan u + \ln |\sec u+\tan u|+c \end{aligned}
Hence
\begin{aligned}
\int \frac{x^2}{\sqrt{x^2+1}} d x&= \int \frac{\tan^2u }{\sqrt{\tan^2u+1}} \sec^2u du\\
&= \int \tan^2 u \sec udu \\
&=\int \left(-1+\sec ^2\left(u\right)\right)\sec \left(u\right)du\\
&=-\int \sec \left(u\right)du+\int \sec ^3\left(u\right)du\\
&= -\frac{1}{2}\ln |\sec u+\tan u | +\frac{1}{2}\sec u \tan u+c\\
&= \frac{1}{2}x\sqrt{x^2+1}-\frac{1}{2}\ln \left|x+\sqrt{x^2+1}\right|+C
\end{aligned}
