Answer
See below
Work Step by Step
$y''-y'-2y=40\sin ^2x$
The complementary function is
$y_c(x)=c_1xe^{-x}+c_2e^{2x}$
Set $\sin^2x=\frac{1}{2}-\frac{1}{2}\cos 2x$
We consider the complex differential equation
$z''-z'-2z=20-20e^{2ix}$
An appropriate complex-valued trial solution for this differential equation is:
$z_p(x)=A_0+B_0e^{2ix}$
where $A_0,B_0$ are complex constants.
so that $z _p$ is a solution if and only if
$-2A_0-(6+2i)B_0e^{2ix}=20-20e^{2ix}\\
A_0=-10,B_0=3-i$
A particular solution is:
$y_p(x)=Re_{\{z_p\}}=3\sin 2x+\cos 2x-10$
so that the general solution is
$y(x)=c_1xe^{-x}+c_2e^{2x}+3\sin 2x+\cos 2x-10$
