Chapter 8 - Linear Differential Equations of Order n - 8.3 The Method of Undetermined Coefficients: Annihilators - Problems - Page 525: 8

$F(x)= 2 G(x)-H(x)$ Therefore, $(D^2-10D+26)^3$ is the annihilator of $F(x)$.

Since, $(D-4)^2$ is the annihilator of $G(x)= xe^{4x}$ Also, we have: $(D^2-10D+26)^3$ is the annihilator of $H(x)=x^{2}e^{5x}\cos x$.This implies that the $(D^2-10D+26)^3$ is the annihilator of $-H(x)=-x^{2}e^{5x}\cos x$ So, $F(x)= 2 G(x)-H(x)$ Therefore, $(D^2-10D+26)^3$ is the annihilator of $F(x)$.