Answer
$y(x)=C_1e^{-x}+C_2e^{-5x}\cos x+C_3e^{-5x}\sin x$
Work Step by Step
Given $y'''+11y''+36y'+26y=0$
Solve the auxiliary equation for the differential equation.
$r^3+11r^2+36r+26=0$
Factor and solve for the roots.
$(r+1)(r^2+10r+26)=0$
Roots are: $r_1=-1$, as a multiplicity of 1 and $r_2=- 5\pm i$ as a multiplicity of 2.
This implies that there are two independent solutions to the differential equation
$y_1(x)=e^{-x}$
$y_2=e^{-5x}\cos x$ and
$y_3(x)=e^{-5x}\sin x$
Therefore, the general equation is equal to $y(x)=C_1e^{-x}+C_2e^{-5x}\cos x+C_3e^{-5x}\sin x$