Answer
a) $a=2, b=-1$
b) $BA=\begin{bmatrix}
1 & 1 &2\\
0& 2 & 2\\
0 & -1 & -1
\end{bmatrix}$
Work Step by Step
a) $A=\begin{bmatrix}
1 & 2 & 3\\
2 & 5 & 7
\end{bmatrix}.\begin{bmatrix}
3& b\\
-4 & a\\
a & b
\end{bmatrix} =\begin{bmatrix}
3a-5 & 2a+4b\\
7a-14& 5a+9b
\end{bmatrix}$
Since $AB=I_2$
$3a-5=1$
$2a+4b=0$
$7a-14=0$
$5a+9b=1$
then we get $a=2, b=-1$
b) Substitute $a=2, b=-1$
then we get: $BA=\begin{bmatrix}
3& -1\\
-4 & 2\\
2 & -1
\end{bmatrix}.\begin{bmatrix}
1 & 2 & 3\\
2 & 5 & 7
\end{bmatrix}=\begin{bmatrix}
1 & 1 &2\\
0& 2 & 2\\
0 & -1 & -1
\end{bmatrix}$
