Answer
$y=c_{1}e^{-x}+c_{2}e^{3x}-\frac{7}{65}cos(2x)-\frac{4}{65}sin(2x)$
Work Step by Step
$y''-2y'-3y=cos(2x)$
Auxiliary equation:
$r^{2}-2r-3=0$
$(r+1)(r-3)=0$
$r_{1}=-1$
$r_{2}=3$
$y_{c}=c_{1}e^{r_{1}x}+c_{2}e^{r_{2}x}$
$y_{c}=c_{1}e^{-1x}+c_{2}e^{3x}$
$y_{p}=Acos(2x)+Bsin(2x)$
$y_{p}'=-2Asin(2x)+2Bcos(2x)$
$y_{p}''=-4Acos(2x)-4Bsin(2x)$
Plug back into the original equation
$(-4Acos(2x)-4Bsin(2x))-2(-2Asin(2x)+2Bcos(2x))-3(Acos(2x)+Bsin(2x))=cos(2x)$
$(-7A-4B)cos(2x)+(4A-7B)sin(2x)=cos(2x)$
thus
$(-7A-4B)=1$
$(4A-7B)=0$
$A=-\frac{7}{65}$
$B=-\frac{4}{65}$
$y=y_{c}+y_{p}$
$y=c_{1}e^{-x}+c_{2}e^{3x}-\frac{7}{65}cos(2x)-\frac{4}{65}sin(2x)$
