Answer
$(Ly)(x)=40x^7+x^4(480-40\cos x)+x^2(480+5\ln x)+\frac{2}{x}+\frac{2}{x^3}-\frac{\cos x}{x}$
Work Step by Step
Given $L=(x^2+1)D^3-(\cos x)D+5x^2$
with $y_1=\ln x+8x^5$
We first compute the appropriate derivatives
$y'(x)=\frac{1}{x}+40x^4\\
y''(x)=-\frac{1}{x^2}+160x^3\\
y'''(x)=\frac{2}{x^3}+480x^2$
Hence
$(Ly)(x)=(\ln x+8x^5)[(x^2+1)D^3-(\cos x)D+5x^2]=(x^2+1)(\frac{2}{x^3}+480x^2)-(\cos x)(\frac{1}{x}+40x^4)+5x^2(\ln x+8x^5)=40x^7+x^4(480-40\cos x)+x^2(480+5\ln x)+\frac{2}{x}+\frac{2}{x^3}-\frac{\cos x}{x}$
