Answer
See below
Work Step by Step
$y''-2y'+10y=24e^{x}\cos 3x$
The complementary function is
$y_c(x)=c_1 e^{x}\sin 3x+c_2 e^{x} \cos 3x$
We consider the complex differential equation
$z''-2z'+10z=24e^{(1+3i)x}$
An appropriate complex-valued trial solution for this differential equation is:
$z_p(x)=A_0xe^{(-1+2i)x}$
where $A_0$ is a complex constant.
so that $z _p$ is a solution if and only if
$6iA_0e^{(1+3i)x}=24e^{(1+3i)x}\\
\rightarrow A_0=-4i$
Hence, $z=-4ixe^{(1+3i)x}$
A particular solution is:
$y_p(x)=Re_{\{z_p\}}=4xe^{x}\sin 3x$
so that the general solution is
$y(x)=c_1e^{x}\sin 3x+c_2e^{x}\cos 3x + 4xe^{x}\sin 3x$
