Answer
$$\int x^3\sqrt{x^2+1}dx=\frac{(x^2+1)^{3/2}(3x^2-2)}{15}+c$$
Work Step by Step
To solve the integral $\int x^3\sqrt{x^2+1}dx$ we will use substitution $x^2+1=t$ which gives us $x^2=t-1$and $2xdx=dt\Rightarrow xdx=\frac{dt}{2}$. Putting this into the integral we get:
$$\int x^3\sqrt{x^2+1}dx=\int \sqrt{x^2+1}\cdot x^2\cdot xdx=
\int\sqrt t(t-1)\frac{dt}{2}=\frac{1}{2}\int t^{3/2}dt-\frac{1}{2}\int t^{1/2}dt=
\frac{1}{2}\frac{t^{5/2}}{\frac{5}{2}}-\frac{1}{2}\frac{t^{3/2}}{\frac{3}{2}}+c=\frac{1}{2}\frac{2}{5}t^{5/2}-\frac{1}{2}\frac{3}{2}t^{3/2}+c=
\frac{1}{5}t^{5/2}-\frac{1}{3}t^{3/2}+c=
\frac{t^{3/2}(3t-5)}{15}+c$$
where $c$ is arbitrary constant.
Now we have to express solution in terms of $x$:
$$\int x^3\sqrt{x^2+1}dx=\frac{t^{3/2}(3t-5)}{15}+c=\frac{(x^2+1)^{3/2}(3(x^2+1)-5)}{15}+c=
\frac{(x^2+1)^{3/2}(3x^2-2)}{15}+c$$
