Answer
See below
Work Step by Step
Given $x^2y''-xy'+5y=8x(\ln x)^2$
In this case the substitution $y(x) = x^r$ yields the indicial equation
$$r(r-1)-r+5=0\\
r^2-2r+5=0$$
It follows that two linearly independent solutions to the given differential equation are
$u_1(x)=\cos (2 \ln x)\\
u_2(x)=\sin (2\ln x)$
so that the general solution is
$y(x)=u_1(x)\cos (2\ln x)+u_2(x)\sin (2\ln x)$
where $u_1$ and $u_2$ are determined from
$u_1'(x)\cos (2\ln x)+u_2'(x)\sin (2\ln x)=0\\
u_1'(x)(\cos (2\ln x))'+u'_2(x)(\sin (2\ln x))'=8x(\ln x)^2$
Hence,
$u_1'=-\frac{4\sin (2\ln x)}{x}(\ln x)^2\\
u_2'=\frac{4\cos (2\ln x)}{x}(\ln x)^2$
which upon integration gives
$u_1(x)=2(\ln x)^2\cos (2\ln x)-2\ln \sin(2\ln x)-\cos (2\ln x)\\
u_2(x)=2(\ln x)^2\sin (2\ln x)-2\ln \cos(2\ln x)+2\sin (2\ln x)$
Thus, the given equation has general solution
$y(x)=c_1\cos (2\ln x)+c_2\sin (2\ln x)+2(\ln x)^2x$
