Answer
See below
Work Step by Step
1. Find eigenvalues:
(A-$\lambda$I)$\vec{V}$=$\vec{0}$
$\begin{bmatrix} 3-\lambda & 0 & 4 \\ 0 & 2-\lambda & 0\\ -4 & 0 & -5-\lambda\end{bmatrix}\begin{bmatrix} v_1\\ v_2 \\ v_3 \end{bmatrix}=\begin{bmatrix} 0\\ 0 \\0 \end{bmatrix}$
$\begin{bmatrix} 3-\lambda & 0 & 4 \\ 0 & 2-\lambda & 0\\ -4 & 0 & -5-\lambda\end{bmatrix}=0$
$(\lambda-2)(\lambda+1)^2=0$
$\lambda_1=2,\lambda_2=\lambda_3=-1$
2. Find eigenvectors:
For $\lambda=2$
let $B=A-\lambda_1I$
$B=\begin{bmatrix} 1 & 0 & 4 \\ 0 & 0 & 0\\ -4 & 0 & -7\end{bmatrix}=\begin{bmatrix} 0\\ 0 \\0 \end{bmatrix} $
Let $r$ be a free variable.
$\vec{V}=r(0,1,0)\\
E_1=\{(0,1,0)\} \\
\rightarrow dim(E_2)=1$
For $\lambda=-1$
let $B=A-\lambda_1I$
$B=\begin{bmatrix} 4 & 0 & 4 \\ 0 & 3 & 0\\ -4 & 0 & -4 \end{bmatrix}=\begin{bmatrix} 0\\ 0 \\0 \end{bmatrix} $
Let $r,s$ be free variables.
$\vec{V}=r(1,0,-1)+s(\frac{1}{4},0,\frac{1}{4})\\
E_2=\{(1,0,-1);(\frac{1}{4},0,\frac{1}{4})\} \\
\rightarrow dim(E_2)=2$
Hence, $S=\begin{bmatrix} 1 & \frac{1}{4} & 0\\ 0 & 0 & 1 \\ -1 & \frac{1}{4} & 0 \end{bmatrix} $
We have $x=Sy \rightarrow x'=Ax \rightarrow y'=\begin{bmatrix} -1 & 1 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 2 \end{bmatrix}y $
then $y_1=c_1e^{-t}+c_2te^{-t}\\
y_2=c_2e^{-t}\\
y_3=c_3e^{2t}$
Hence,
$x(t)=S.y(t)=\begin{bmatrix} 1 & \frac{1}{4} & 0\\ 0 & 0 & 1\\ 1 & \frac{1}{4} & 0 \end{bmatrix} \begin{bmatrix} c_1e^{-t}+c_2te^{-t}\\ c_2e^{-t} \\ c_3e^{2t} \end{bmatrix} \\
=\begin{bmatrix} c_1e^{-t}+c_2te^{-t}+\frac{1}{4}c_2e^{-t}\\ c_3e^{2t}\\ c_1e^{-t}+c_2te^{-t} +\frac{1}{4}c_2e^{-t}\end{bmatrix} $