Answer
See below
Work Step by Step
Given: $y''-2y'+26y=e^x \cos 5x$
In this case the substitution $y(x) = x^r$ yields the indicial equation
$r^2-2r+26=0$
Solve the equation
$r_1=1 + 5i\\
r_2=1-5i$
It follows that two linearly independent solutions to the given differential equation are
$y_1(x)=e^{x}\sin 5x\\
y_2(x)=e^{x}\cos 5x$
so that the particular solution is
$y(x)=u_1e^{x}\sin 5x+u_2e^{x}\cos 5x$
We first compute the appropriate derivatives
$0=u_1'(x)e^{x}\sin 5x+u_2'(x)e^{x}\cos 5x\\
e^x\cos 5x=u_1'(x)(e^{x}\sin 5x)'+u_2'(x)(e^{x}\cos 5x)'$
Substituting these expressions into the given equation, we find that $u$ must satisfy
$u_1'=\cos^2 5x\\
u_1=\frac{1}{2}x+\frac{1}{20}\sin 10x$
and $u_2'=-\sin 5x+\cos 5x\\
u_2=\frac{1}{20}\cos 10x$
Hence, the general solution is:
$y(x)=\frac{x}{2}e^{x}\sin 5x+\frac{1}{20}(\sin 5x\sin 10x+\cos 5x \cos 10x)$
