Answer
$$\frac{9}{{10}}{\cos ^{ - 1}}\left( {\frac{5}{{3x}}} \right) - \frac{{\sqrt {9{x^2} - 25} }}{{2{x^2}}} + C$$
Work Step by Step
$$\eqalign{
& \int {\frac{{\sqrt {9{x^2} - 25} }}{{{x^3}}}dx} \cr
& {\text{substitute }}x = \frac{5}{3}sec\theta ,{\text{ }}dx = \frac{5}{3}\sec \theta \tan \theta d\theta \cr
& \int {\frac{{\sqrt {9{x^2} - 25} }}{{{x^3}}}dx} = \int {\frac{{\sqrt {9{{\left( {5/3\sec \theta } \right)}^2} - 25} }}{{{{\left( {\left( {5/3} \right)sec\theta } \right)}^3}}}\left( {\frac{5}{3}\sec \theta \tan \theta d\theta } \right)} \cr
& {\text{simplify}} \cr
& = \int {\frac{{27\sqrt {25{{\sec }^2}\theta - 25} }}{{125{{\sec }^3}\theta }}\left( {\frac{5}{3}\sec \theta \tan \theta d\theta } \right)} \cr
& = \int {\frac{{27\left( 5 \right)\sqrt {{{\sec }^2}\theta - 1} }}{{125{{\sec }^3}\theta }}\left( {\frac{5}{3}\sec \theta \tan \theta d\theta } \right)} \cr
& {\text{pythagorean identity}} \cr
& = \frac{9}{5}\int {\frac{{\sqrt {{{\tan }^2}\theta } }}{{{{\sec }^3}\theta }}\left( {\sec \theta \tan \theta d\theta } \right)} \cr
& = \frac{9}{5}\int {\frac{{{{\tan }^2}\theta }}{{{{\sec }^2}\theta }}d\theta = } \frac{9}{5}\int {\frac{{{{\sin }^2}\theta }}{{{{\cos }^2}\theta }}\left( {{{\cos }^2}\theta } \right)d\theta } \cr
& = \frac{9}{5}\int {{{\sin }^2}\theta d\theta } \cr
& {\text{half - angle formula}} \cr
& = \frac{9}{5}\int {\left( {\frac{{1 - \cos 2\theta }}{2}} \right)d\theta } \cr
& {\text{integrating}} \cr
& = \frac{9}{{10}}\left( {\theta - \frac{1}{2}\sin 2\theta } \right) + C \cr
& = \frac{9}{{10}}\theta - \frac{9}{{10}}\sin \theta \cos \theta + C \cr
& = \frac{9}{{10}}{\sec ^{ - 1}}\left( {\frac{{3x}}{5}} \right) - \frac{9}{{10}}\left( {\frac{{\sqrt {9{x^2} - 25} }}{{3x}}} \right)\left( {\frac{5}{{3x}}} \right) + C \cr
& = \frac{9}{{10}}{\sec ^{ - 1}}\left( {\frac{{3x}}{5}} \right) - \frac{{\sqrt {9{x^2} - 25} }}{{2{x^2}}} + C \cr
& or \cr
& = \frac{9}{{10}}{\cos ^{ - 1}}\left( {\frac{5}{{3x}}} \right) - \frac{{\sqrt {9{x^2} - 25} }}{{2{x^2}}} + C \cr} $$
