Answer
$\frac{1}{7}{\sec ^7}x - \frac{2}{5}{\sec ^5}x + \frac{1}{3}{\sec ^3}x + C$
Work Step by Step
$$\eqalign{
& \text{Let }I=\int {{{\tan }^5}x{{\sec }^3}x} dx \cr
& {\text{Split }}{\tan ^5}x{\text{ as }}{\tan ^4}x\tan x{\text{ and }}{\sec ^3}x{\text{ as }}{\sec ^2}x\sec x \cr
& I = \int {\left( {{{\tan }^4}x\tan x} \right)\left( {{{\sec }^2}x\sec x} \right)} dx \cr
& = \int {{{\tan }^4}x{{\sec }^2}x} \left( {\sec x\tan x} \right)dx \cr
& {\text{Use the property }}{\left( {{a^m}} \right)^n} = {a^{mn}} \cr
& I= \int {{{\left( {{{\tan }^2}x} \right)}^2}{{\sec }^2}x} \left( {\sec x\tan x} \right)dx \cr
& {\text{Rewrite }}{\tan ^2}x{\text{ using }}{\sec ^2}\theta = 1 + {\tan ^2}\theta \cr
& I= \int {{{\left( {{{\sec }^2}x - 1} \right)}^2}{{\sec }^2}x} \left( {\sec x\tan x} \right)dx \cr
& {\text{Expand the binomial}} \cr
& I= \int {\left( {{{\sec }^4}x - 2{{\sec }^2}x + 1} \right){{\sec }^2}x} \left( {\sec x\tan x} \right)dx \cr
& = \int {\left( {{{\sec }^6}x - 2{{\sec }^4}x + {{\sec }^2}x} \right)} \left( {\sec x\tan x} \right)dx \cr
& {\text{Integrate by the substitution method}} \cr
& {\text{Let }}u = \sec x,{\text{ }}du = \sec x\tan xdx \cr
& {\text{Substituting}} \cr
& I= \int {\left( {{u^6} - 2{u^4} + {u^2}} \right)} du \cr
& {\text{Use the power rule }}\int {{u^n}} du = \frac{1}{{n + 1}}{u^{n + 1}} + C \cr
& I= \frac{1}{7}{u^7} - \frac{2}{5}{u^5} + \frac{1}{3}{u^3} + C \cr
& {\text{Write in terms of }}x,{\text{ substitute }}\sec x{\text{ for }}u \cr
& I= \frac{1}{7}{\sec ^7}x - \frac{2}{5}{\sec ^5}x + \frac{1}{3}{\sec ^3}x + C \cr} $$
