Answer
$y_p(x)=xe^{-x}[(A x^2+Bx+C)\cos 3x+(D x^2+Ex+F)\sin 3x]$
Work Step by Step
Consider $G(x)=e^{ax} A(x) \sin mx $ or $G(x)=e^{ax} A(x) \cos mx $
The trial solution for the method of undetermined coefficients can be calculated as:
$y_p(x)=e^{ax} B(x) \sin mx +e^{ax} C(x) \cos mx$
when the sum of the coefficients of a differential equation is zero.
Then, we have $y_p(x)=x e^{ax} B(x)$
Given: $y''+2y'+10y=x^2e^{-x}\cos 3x$
Here, we have $m=k=1$
Thus, the trial solution for the method of undetermined coefficients is:
$y_p(x)=xe^{-x}(A x^2+Bx+C)\cos 3x+xe^{-x}(D x^2+Ex+F)\sin 3x$
Hence, $y_p(x)=xe^{-x}[(A x^2+Bx+C)\cos 3x+(D x^2+Ex+F)\sin 3x]$