Answer
(a) $AB$ not defined.
(b) $$BA=\left[ \begin {array}{cc} 3&-4\\ 10&16
\\26&46\end {array} \right].$$
Work Step by Step
Given
$$
A=\left[\begin{array}{rr}{2} & {1} \\ {-3} & {4} \\ {1} & {6}\end{array}\right], \quad B=\left[\begin{array}{rrr}{0} & {-1} & {0} \\ {4} & {0} & {2} \\ {8} & {-1} & {7}\end{array}\right]
$$
We have
(a) $AB$ not defined.
(b) \begin{align*}
BA&=\left[\begin{array}{rrr}{0} & {-1} & {0} \\ {4} & {0} & {2} \\ {8} & {-1} & {7}\end{array}\right]\left[\begin{array}{rr}{2} & {1} \\ {-3} & {4} \\ {1} & {6}\end{array}\right]\\
&=\left[ \begin {array}{cc} 3&-4\\ 10&16
\\26&46\end {array} \right].
\end{align*}
