Answer
$b=\frac{-5}{13}$
$ a=\frac{12}{13}$
Work Step by Step
Let $A$ be a $m\times n$ matrix corresponding to the linear transformation, say $T$. Then $T$ is defined by $T(v)=Av$, for all $v$ in the given domain.
Here, $T:R^2\rightarrow R^2$ and the corresponding matrix is given by:
$A = \left[ {\begin{array}{*{20}{c}}
a&{ - b}\\
b&a
\end{array}} \right]$
Now,
$\begin{array}{l}
T\left[ {\begin{array}{*{20}{c}}
{12}\\
5
\end{array}} \right] = \left[ {\begin{array}{*{20}{c}}
{13}\\
0
\end{array}} \right]\\
\Rightarrow \left[ {\begin{array}{*{20}{c}}
a&{ - b}\\
b&a
\end{array}} \right]\left[ {\begin{array}{*{20}{c}}
{12}\\
5
\end{array}} \right] = \left[ {\begin{array}{*{20}{c}}
{13}\\
0
\end{array}} \right]
\end{array}$
$ \Rightarrow 12a-5b=13, (1)\\
\quad 12b+5a=0,(2)$
By equation $(2)$, $\displaystyle a=\frac{-12b}{5}$
Puting this in equation $(1)$, we have
$\displaystyle 12(\frac{-12b}{5})-5b=13$
$\Rightarrow b=\frac{-5}{13}$
$\Rightarrow a=\frac{12}{13}$
