Answer
$$\frac{{2{\pi ^{3/2}}}}{3}\left( {\frac{1}{{3\sqrt 3 }} - \frac{1}{8}} \right)$$
Work Step by Step
$$\eqalign{
& \int_1^{\sqrt 3 } {\frac{{\sqrt {{{\tan }^{ - 1}}x} }}{{1 + {x^2}}}} dx \cr
& {\text{substitute }}u = {\tan ^{ - 1}}x,{\text{ }}du = \frac{1}{{1 + {x^2}}}dx \cr
& {\text{With this substitution}}{\text{,}} \cr
& {\text{if }}x = \sqrt 3 ,{\text{ }}u = {\tan ^{ - 1}}\left( {\sqrt 3 } \right) = \frac{\pi }{3} \cr
& {\text{if }}x = 1,{\text{ }}u = {\tan ^{ - 1}}\left( 1 \right) = \frac{\pi }{4} \cr
& {\text{so}} \cr
& \int_1^{\sqrt 3 } {\frac{{\sqrt {{{\tan }^{ - 1}}x} }}{{1 + {x^2}}}} dx = \int_{\pi /4}^{\pi /3} {\sqrt u } du \cr
& = \int_{\pi /4}^{\pi /3} {{u^{1/2}}} du \cr
& {\text{find the antiderivative}} \cr
& = \left. {\left( {\frac{{2{u^{3/2}}}}{3}} \right)} \right|_{\pi /4}^{\pi /3} \cr
& {\text{part 1 of fundamental theorem of calculus}} \cr
& = \frac{{2{{\left( {\pi /3} \right)}^{3/2}}}}{3} - \frac{{2{{\left( {\pi /4} \right)}^{3/2}}}}{3} \cr
& {\text{simplify}} \cr
& = \frac{{2{\pi ^{3/2}}}}{3}\left( {\frac{1}{{{3^{3/2}}}}} \right) - \frac{{2{\pi ^{3/2}}}}{3}\left( {\frac{1}{8}} \right) \cr
& {\text{factor out }}\frac{{2{\pi ^{3/2}}}}{3} \cr
& = \frac{{2{\pi ^{3/2}}}}{3}\left( {\frac{1}{{3\sqrt 3 }} - \frac{1}{8}} \right) \cr} $$
