Answer
$\displaystyle \frac{\sin^{6}x}{6}-\frac{\sin^{8}x}{8}+C$
The integrand, $f(x) $
(red graph) behaves as $F'(x)$ so the answer is reasonable.
Work Step by Step
From the Strategy for Evaluating $\displaystyle \int\sin^{m}x\cos^{n}xdx$, case (a)
(the power of cosine is odd)
we use the identity $\sin^{2}x=1-\cos^{2}x$ and then substitute $u=\sin x$:
$\displaystyle \int\sin^{5}x\cos^{3}xdx=\int\sin^{5}x\cos^{2}x\cos xdx$
$=\displaystyle \int\sin^{5}x(1-\sin^{2}x)\cos xdx$
$=\displaystyle \int u^{5}(1-u^{2})du$
$=\displaystyle \int(u^{5}-u^{7})du$
$=\displaystyle \frac{u^{6}}{6}-\frac{u^{8}}{8}+C$
... bring back x,
$=\displaystyle \frac{\sin^{6}x}{6}-\frac{\sin^{8}x}{8}+C$
Graphing, the integrand $f(x)=\sin^{5}x\cos^{3}x$
should behave as $F'(x)$, where $F(x)=\displaystyle \frac{\sin^{6}x}{6}-\frac{\sin^{8}x}{8}$
In the intervals where $F$ decreases , $f$ is negative.
When $F$ increases , $f$ is positive.
At the minimum points of F, f is zero, changing from negative to positive.
At the maximum points of F, f is zero, changing from positive to negative.
$f(x)$(red graph) behaves as $F'(x)$ so the answer is reasonable.
