Answer
$$\frac{2}{15} (x^{3}+1)^{5 / 2}-\frac{2}{9} (x^{3}+1)^{3 / 2}+c$$
Work Step by Step
Given $$ \int x^{5} \sqrt{x^{3}+1} d x $$
Let
$$u=x^{3}+1 \ \ \ \Rightarrow \ \ du=3x^2dx $$
then
\begin{align*}
\int x^{5} \sqrt{x^{3}+1} d x &=\frac{1}{3} \int(u-1) \sqrt{u} d u \\
&=\frac{1}{3} \int(u-1) u^{1 / 2} d u \\
&=\frac{1}{3} \int\left(u^{3 / 2}-u^{1 / 2}\right) d u \\ &=\frac{1}{3}\left(\frac{2}{5} u^{5 / 2}-\frac{2}{3} u^{3 / 2}+c\right) \\
&=\frac{2}{15} u^{5 / 2}-\frac{2}{9} u^{3 / 2}+c\\
&= \frac{2}{15} (x^{3}+1)^{5 / 2}-\frac{2}{9} (x^{3}+1)^{3 / 2}+c
\end{align*}
