Answer
See below
Work Step by Step
Given: $y'''+y''-10y'+8y=0$
Substitute: $y'''(x)+y''(x)-10y'(x)+8y(x)\\
=r^3e^{rx}+r^2e^{rx}-10re^{rx}+8re^{rx}\\
=e^{rx}(r^3-r^2-10r+8)\\
=0$
Since $e^{rx} \ne0 \rightarrow r^3-r^2-10r+8=0\\
\rightarrow (r-2)(r-1)(r+4)=0\\
\rightarrow r=-4, r=1,r=2$
The solutions to the given problem are: $y_1(x)=e^{-4x}\\
y_2(x)=e^{x} \\
y_3(x)=e^{2x}$
Obtain: $W_{[y_1,y_2,y_3]}(x)=\begin{vmatrix}
e^{-4x} & e^{x} & e^{2x}\\
-4e^{-4x} & e^{x} & 2e^{2x} \\
16e^{-4x} & e^{x} & 4e^{2x}
\end{vmatrix}=e^{-4x}\begin{vmatrix}
e^{x} & 2e^{2x}\\
e^{x} & 4e^{2x}
\end{vmatrix} -e^x\begin{vmatrix}
-4e^{-4x} & 2e^{2x}\\
16e^{-4x} & 4e^{2x}
\end{vmatrix} + 2e^{2x}\begin{vmatrix}
-4e^{-4x} & e^{x}\\
16e^{-4x} & -e^{x}
\end{vmatrix}=2e^{-x}+48e^{-x}-20e^{-x}=30e^{-x}$
Since $30e^{-x} \ne0$, $y_1,y_2,y_3$ are linearly independent solutions to the equation.
Hence, the solutions is $y(x)=C_1e^{-4x}+C_2e^{x}+C_3e^{2x}$
