Answer
$\pi a^2 b$
Work Step by Step
Consider $I=\iint_{D} (ax^3+by^3+\sqrt {a^2-x^2}) d A$
or, $=\int_{-a}^a \int_{-b}^b (ax^3+by^3+\sqrt {a^2-x^2}) dy \ dx $
or, $=\int_{-a}^a (ayx^3+\dfrac{by^4}{4}+y\sqrt {a^2-x^2})_{-b}^b \ dx $
or, $= 2b \int_{-a}^a ax^3 \ dx + 2b \int_{-a}^a \sqrt {a^2-x^2} \ dx $
or, $=2b \int_{-a}^a \sqrt {a^2-x^2} \ dx $
or, $=2b [\dfrac{u\sqrt {a^2-u^2}}{2} +\dfrac{a^2}{2} \sin^{-1} \dfrac{u}{a}]_{-a}^{a}$
or, $=4b[\dfrac{a\sqrt {a^2-a^2}}{2} +\dfrac{a^2}{2} \sin^{-1} \dfrac{a}{a}]$
or, $=2a^2b \sin^{-1} (1) $
or, $=2a^2b \times \dfrac{\pi}{2}$
or, $=\pi a^2 b$