Answer
$$\displaystyle{\int\frac{\sqrt{x^2-9}}{x^3}dx=\frac{\arccos\left(\frac{3}{x}\right)}{6}-\frac{\sqrt{x^2-9}}{2x^2}+C}\\$$
Work Step by Step
$\displaystyle{I=\int\frac{\sqrt{x^2-9}}{x^3}dx}\\$
$\displaystyle \left[\begin{array}{ll} x=3\sec\theta & x^2=9\sec^2\theta \\ & \\ \frac{dx}{d\theta}=3\sec\theta\tan\theta & dx=3
\sec\theta\tan\theta\ d\theta \end{array}\right]$ Integration by substitution
$\displaystyle{I=\int\frac{\sqrt{9\sec^2\theta-9}}{27\sec^3 \theta}3\sec\theta\tan\theta\ d\theta}\\
\displaystyle{I=\int\frac{\sqrt{9\left(\sec^2\theta-1\right)}}{9\sec^2 \theta}\tan\theta\ d\theta}\\
\displaystyle{I=\int\frac{3\tan\theta}{9\sec^2 \theta}\tan\theta\ d\theta}\\
\displaystyle{I=\frac{1}{3}\int\frac{\tan^2\theta}{\sec^2 \theta}\ d\theta}\\
\displaystyle{I=\frac{1}{3}\int{\sin^2\theta}\ d\theta}\\
\displaystyle{I=\frac{1}{3}\int\frac{1}{2}\left(1-\cos2\theta\right)\ d\theta}\\
\displaystyle{I=\frac{1}{6}\int1-\cos2 \theta\ d\theta}\\
\displaystyle{I=\frac{1}{6}\theta-\frac{1}{12}\sin2\theta+C}\\
\displaystyle{I=\frac{1}{6}\theta-\frac{1}{12}2\sin\theta\cos\theta+C}\\
\displaystyle{I=\frac{1}{6}\theta-\frac{1}{6}\sin\theta\cos\theta+C}\\$
$\sin\theta=\frac{\sqrt{x^2-9}}{x}\\
\cos\theta=\frac{3}{x}\\
\theta=\arccos\left(\frac{3}{x}\right)\\$
$\displaystyle{I=\frac{1}{6}\arccos\left(\frac{3}{x}\right)-\frac{1}{6}\frac{\sqrt{x^2-9}}{x}\times\frac{3}{x}+C}\\
\displaystyle{I=\frac{\arccos\left(\frac{3}{x}\right)}{6}-\frac{\sqrt{x^2-9}}{2x^2}+C}\\$
