Chapter 8 - Linear Differential Equations of Order n - 8.3 The Method of Undetermined Coefficients: Annihilators - Problems - Page 526: 41

See below

We can notice that: $F(x)=5e^{-2x}\cos 3x $ Obtain: $(D^2+4D+13)^2y_p(x)= 5e^{-2x}\cos 3x $ Therefore, the general solution for the given differential equation is: $y(x)=c_1e^{-2x}\cos 3x+c_2 e^{-2x} \sin 3x+x^2A_0e^{-2x}\cos 3x+x^2B_0e^{-2x}\sin 3x$ The trial solution is $y_p=x^2A_0e^{-2x}\cos 3x+x^2B_0e^{-2x}\sin 3x$.