Answer
$y_p(x)=(Ax+B)e^{x} \sin x +(Cx+D)e^{x} \cos x$
Work Step by Step
Consider $G(x)=e^{ax} A(x) \sin mx $ or $G(x)=e^{ax} A(x) \cos mx $
The trial solution for the method of undetermined coefficients can be calculated as:
$y_p(x)=e^{ax} B(x) \sin mx +e^{ax} C(x) \cos mx$
Given: $y''-y'-2y=xe^x \cos x$
Here, we have $m=k=1$ . The degree of the polynomials $B(x) ; C(x)=1$
Thus, the trial solution for the method of undetermined coefficients is:
$y_p(x)=(Ax+B)e^{x} \sin x +(Cx+D)e^{x} \cos x$
