Answer
\[\frac{\sec^7x}{7}+\frac{\sec^3x}{3}-\frac{2\sec^5x}{5}+C\]
Work Step by Step
\[I=\int\tan^5x\:\sec^3xdx\]
$I=\int\tan^4x\:\sec^2x(\sec x\:\tan x)dx$
$[\sec^2 x-\tan^2 x=1]$
$I=\int(\sec^2 x-1)^2\:\sec^2x(\sec x\:\tan x)dx$
Substitute $\sec x=t$
$\Rightarrow \sec x\:\tan xdx=dt$
$I=\int(t^2-1)^2t^2dt$
$I=\int(t^4+1-2t^2)t^2dt$
$I=\int(t^6+t^2-2t^4)dt$
$I=\frac{t^7}{7}+\frac{t^3}{3}-\frac{2t^5}{5}+C$
Where $C$ is constant of integration
$I=\large\frac{\sec^7x}{7}+\frac{\sec^3x}{3}-\frac{2\sec^5x}{5}$+$C$
Hence $I=\large\frac{\sec^7x}{7}+\frac{\sec^3x}{3}-\frac{2\sec^5x}{5}$+$C$.
