Answer
$2a^3b\sqrt[]{ab}$
Work Step by Step
Using the properties of radicals, then,
\begin{array}{l}
a^3\sqrt[]{9ab^3}-\sqrt[]{25a^7b^3}+\sqrt[]{16a^7b^3}
\\=
a^3\sqrt[]{9b^2\cdot ab}-\sqrt[]{25a^6b^2\cdot ab}+\sqrt[]{16a^6b^2\cdot ab}
\\=
a^3\cdot3b\sqrt[]{ab}-5a^3b\sqrt[]{ab}+4a^3b\sqrt[]{ab}
\\=
3a^3b\sqrt[]{ab}-5a^3b\sqrt[]{ab}+4a^3b\sqrt[]{ab}
\\=
2a^3b\sqrt[]{ab}
.\end{array}