Answer
$a^2b+ab^2$
Work Step by Step
By using the cofactor expansion along the first row:
$det\begin{vmatrix}
a& b&0\\
0&a &b\\
a&0 &b
\end{vmatrix}$
$=a\times det\begin{vmatrix}
a&b\\
0 &b
\end{vmatrix}-b\times det\begin{vmatrix}
0 &b\\
a&b
\end{vmatrix}+0\times det\begin{vmatrix}
0&a\\
a&0
\end{vmatrix}$
$=a(ab-0)-b(0-ab)+0$
$=a^2b+ab^2$