Answer
$$\frac{{154}}{5}\sqrt 7 $$
Work Step by Step
$$\eqalign{
& \int_3^4 {{x^3}\sqrt {{x^2} - 9} } dx \cr
& {\text{Let }}x = 3\sec \theta ,{\text{ }}dx = 3\sec \theta \tan \theta d\theta \cr
& \theta = {\sec ^{ - 1}}\frac{x}{3} \cr
& {\text{The new limits of integration are:}} \cr
& x = 4 \to \theta = {\sec ^{ - 1}}\frac{4}{3} \cr
& x = 3 \to \theta = {\sec ^{ - 1}}1 = 0 \cr
& {\text{Substituting}} \cr
& \int_3^4 {{x^3}\sqrt {{x^2} - 9} } dx = \int_0^{{{\sec }^{ - 1}}\frac{4}{3}} {{{\left( {3\sec \theta } \right)}^3}\sqrt {{{\left( {3\sec \theta } \right)}^2} - 9} } \left( {3\sec \theta \tan \theta } \right)d\theta \cr
& = {3^4}\int_0^{{{\sec }^{ - 1}}\frac{4}{3}} {{{\sec }^3}\theta \sqrt {9{{\sec }^2}\theta - 9} } \left( {\sec \theta \tan \theta } \right)d\theta \cr
& = {3^5}\int_0^{{{\sec }^{ - 1}}\frac{4}{3}} {{{\sec }^3}\theta \sqrt {{{\sec }^2}\theta - 1} } \left( {\sec \theta \tan \theta } \right)d\theta \cr
& = {3^5}\int_0^{{{\sec }^{ - 1}}\frac{4}{3}} {{{\sec }^3}\theta \left( {\tan \theta } \right)} \left( {\sec \theta \tan \theta } \right)d\theta \cr
& = {3^5}\int_0^{{{\sec }^{ - 1}}\frac{4}{3}} {{{\sec }^4}\theta {{\tan }^2}\theta } d\theta \cr
& = {3^5}\int_0^{{{\sec }^{ - 1}}\frac{4}{3}} {{{\sec }^2}\theta {{\tan }^2}\theta {{\sec }^2}\theta } d\theta \cr
& = {3^5}\int_0^{{{\sec }^{ - 1}}\frac{4}{3}} {\left( {1 + {{\tan }^2}\theta } \right){{\tan }^2}\theta {{\sec }^2}\theta } d\theta \cr
& = 243\int_0^{{{\sec }^{ - 1}}\frac{4}{3}} {\left( {{{\tan }^2}\theta + {{\tan }^4}\theta } \right){{\sec }^2}\theta } d\theta \cr
& {\text{Integrating}} \cr
& {\text{ = 243}}\left[ {\frac{1}{3}{{\tan }^3}\theta + \frac{1}{5}{{\tan }^5}\theta } \right]_0^{{{\sec }^{ - 1}}\frac{4}{3}} \cr
& {\text{ = }}\frac{{{\text{81}}}}{5}\left[ {5{{\tan }^3}\theta + 3{{\tan }^5}\theta } \right]_0^{{{\sec }^{ - 1}}\frac{4}{3}} \cr
& {\text{ = }}\frac{{{\text{81}}}}{5}\left[ {5{{\tan }^3}\left( {{{\sec }^{ - 1}}\frac{4}{3}} \right) + 3{{\tan }^5}\left( {{{\sec }^{ - 1}}\frac{4}{3}} \right)} \right] - \frac{{{\text{81}}}}{5}\left[ 0 \right] \cr
& {\text{Where }}\tan \left( {{{\sec }^{ - 1}}\frac{4}{3}} \right) = \frac{{\sqrt 7 }}{3} \cr
& {\text{ = }}\frac{{{\text{81}}}}{5}\left[ {5{{\left( {\frac{{\sqrt 7 }}{3}} \right)}^3} + 3{{\left( {\frac{{\sqrt 7 }}{3}} \right)}^5}} \right] \cr
& {\text{ = }}\frac{{{\text{81}}}}{5}\left[ {\frac{{35\sqrt 7 }}{{27}} + \frac{{49\sqrt 7 }}{{81}}} \right] \cr
& {\text{ = }}\frac{{{\text{81}}}}{5}\left( {\frac{{154}}{{81}}\sqrt 7 } \right) \cr
& {\text{ = }}\frac{{154}}{5}\sqrt 7 \cr} $$
