Answer
\[\frac{\sec^7 \theta}{7}+\frac{\sec^3 \theta}{3}-\frac{2}{5}\sec^5 \theta+C\]
Work Step by Step
Let \[I=\int\tan^5 \theta\;\sec^3 \theta\;d \theta\]
\[I=\int\tan^4 \theta\;\sec^2 \theta(\sec \theta\tan\theta)\;d\theta\]
\[\left[\tan^2\theta=\sec^2\theta-1\right]\]
\[I=\int\left(\sec^2 \theta-1\right)^2\;\sec^2 \theta(\sec \theta\tan\theta)\;d\theta\]
Substitute $\;t=\sec\theta\;\;\;\ldots (1)$
\[\Rightarrow dt=\sec\theta\tan\theta\;d\theta\]
\[I=\int\left(t^2-1\right)^2t^2\;dt\]
\[I=\int\left(t^4+1-2t^2\right)t^2\:dt\]
\[I=\int\left(t^6+t^2-2t^4\right)dt\]
\[I=\frac{t^7}{7}+\frac{t^3}{3}-\frac{2}{5}t^5+C\]
Where $C$ is constant of integration
From (1)
\[I=\frac{\sec^7 \theta}{7}+\frac{\sec^3 \theta}{3}-\frac{2}{5}\sec^5 \theta+C\]
Hence,
\[I=\frac{\sec^7 \theta}{7}+\frac{\sec^3 \theta}{3}-\frac{2}{5}\sec^5 \theta+C \;\;.\]
