Chapter 5 - Section 5.5 - Indefinite Integrals and the Substitution Method - Exercises - Page 330: 64

$$\int \frac{\sqrt{\tan^{-1}x}}{1+x^2}dx=\frac{2}{3}(\tan^{-1}x)^{3/2}+C$$

$$A=\int \frac{\sqrt{\tan^{-1}x}}{1+x^2}dx$$ We set $u=\tan^{-1}x$ Then $$du=\frac{1}{1+x^2}dx$$ Therefore, $$A=\int \sqrt udu=\int u^{1/2}du$$ $$A=\frac{u^{3/2}}{\frac{3}{2}}+C=\frac{2u^{3/2}}{3}+C$$ $$A=\frac{2}{3}(\tan^{-1}x)^{3/2}+C$$