Answer
$$\int 3x^5\sqrt{x^3+1}dx=\frac{2}{5}(x^3+1)^{5/2}-\frac{2}{3}(x^3+1)^{3/2}+C$$
Work Step by Step
$$A=\int 3x^5\sqrt{x^3+1}dx=\int (x^3\sqrt{x^3+1})3x^2dx$$
We set $u=x^3+1$, which means $x^3=u-1$
Then $$du=3x^2dx$$
Therefore, $$A=\int(u-1)\sqrt udu=\int\Big((u-1)u^{1/2}\Big)du$$ $$A=\int\Big(u^{3/2}-u^{1/2}\Big)du$$ $$A=\Big(\frac{u^{5/2}}{\frac{5}{2}}-\frac{u^{3/2}}{\frac{3}{2}}\Big)+C$$ $$A=\frac{2u^{5/2}}{5}-\frac{2u^{3/2}}{3}+C$$ $$A=\frac{2}{5}(x^3+1)^{5/2}-\frac{2}{3}(x^3+1)^{3/2}+C$$
