Answer
$$\frac{2}{3}x\sqrt x \arctan \left( {x\sqrt x } \right) - \frac{2}{3}\ln \sqrt {1 + {x^3}} + C$$
Work Step by Step
$$\eqalign{
& \int {\sqrt x \arctan {x^{3/2}}} dx \cr
& {\text{Let }}u = {x^{3/2}}.{\text{ Then }}du = \frac{3}{2}{x^{1/2}}dx,{\text{ }}dx = \frac{2}{{3{x^{1/2}}}}du \cr
& {\text{Substituting you have}} \cr
& \int {\sqrt x \arctan {x^{3/2}}} dx = \int {\sqrt x \arctan u} \left( {\frac{2}{{3{x^{1/2}}}}du} \right) \cr
& = \frac{2}{3}\int {\arctan u} du \cr
& {\text{From the table of integrals in the back of the book}} \cr
& \int {\arctan u} du = u\arctan u - \ln \sqrt {1 + {u^2}} + C \cr
& \frac{2}{3}\int {\arctan u} du = \frac{2}{3}u\arctan u - \frac{2}{3}\ln \sqrt {1 + {u^2}} + C \cr
& {\text{Write in terms of }}x,{\text{ }}u = {x^{3/2}} \cr
& = \frac{2}{3}{x^{3/2}}\arctan {x^{3/2}} - \frac{2}{3}\ln \sqrt {1 + {{\left( {{x^{3/2}}} \right)}^2}} + C \cr
& = \frac{2}{3}x\sqrt x \arctan \left( {x\sqrt x } \right) - \frac{2}{3}\ln \sqrt {1 + {x^3}} + C \cr} $$
