Answer
See below
Work Step by Step
Given: $y'''+3y''-18y'-40y=0$
Substitute: $y'''(x)+3y''(x)-18y'(x)-40y(x)\\
=r^3e^{rx}+3r^2e^{rx}-18re^{rx}-40e^{rx}\\
=e^{rx}(r^3+3r^2-18r-40)\\
=0$
Since $e^{rx} \ne0 \rightarrow r^3+3r^2-18r-40=0\\
\rightarrow (r+5)(r+2)(r-4)=0\\
\rightarrow r=4, r=-2,r=-5$
The solutions to the given problem are: $y_1(x)=e^{-5x} \\
y_2(x)=e^{-2x} \\
y_3(x)=e^{4x}$
Obtain: $W_{[y_1,y_2]}(x)=\begin{vmatrix}
e^{-5x} & e^{-2x} & e^{4x}\\
-5e^{-5x} & -2e^{-2x} & 4e^{4x} \\
25e^{-5x} & 4e^{-2x} & 16e^{4x}
\end{vmatrix}=e^{-5x}\begin{vmatrix}
-2e^{-2x} & 4e^{4x}\\
4e^{-2x} & 16e^{4x}
\end{vmatrix} - e^{-2x}\begin{vmatrix}
-5e^{-5x} & 4e^{4x}\\
25e^{-5x} & 16e^{4x}
\end{vmatrix} + e^{4x}\begin{vmatrix}
-5e^{-5x} & -2e^{-2x}\\
25e^{-5x} & 4e^{-2x}
\end{vmatrix}=-48e^{-3x}+180e^{-3x}+30e^{-3x}=162e^{-3x}$
Since $162e^{-3x} \ne0$, $y_1,y_2,y_3$ are linearly independent solutions to the equation.
Hence, the solutions is $y(x)=C_1e^{-5x}+C_2e^{-2x}+C_3e^{4x}$
