Answer
See below
Work Step by Step
Given: $y''-9y'+20y=x^3e^{5x}$
In this case the substitution $y(x) = x^r$ yields the indicial equation
$r^2-9r+20=0$
Solve the equation
$r_1=4\\
r_2=5$
It follows that two linearly independent solutions to the given differential equation are
$y_1(x)=e^{4x}\\
y_2(x)=e^{5x}$
so that the particular solution is
$y(x)=u_1e^{4x}+u_2e^{5x}$
We first compute the appropriate derivatives
$0=u_1'(x)e^{4x}+u_2'(x)e^{5x}\\
x^3e^{5x}=u_1'(x)(e^{4x})'+u_2'(x)(e^{5x})'$
Substituting these expressions into the given equation, we find that $u$ must satisfy
$u_1'=-x^3e^x\\
u_1=4x^2e^x+e^x-6xe^x-x^3e^x$
and $u_2'=x^3\\
u_2=\frac{x^4}{4}$
Hence, the general solution is:
$y(x)=\frac{x^4}{4}e^{5x}-x^3e^{5x}+3x^2e^{5x}+e^{5x}$
