Answer
$\iint_D xy\sqrt{1+x^2+y^2}dA=\int_0^1 \frac{r^2\sqrt{1+r^2}}{2}dr\approx 0.2101$
Work Step by Step
$D=\{(r,\theta)|0\leq r\leq 1,0\leq \theta\leq \frac{\pi}{2}\}$
$\iint_Dxy\sqrt{1+x^2+y^2}dA=\int_0^1\int_0^{\pi/2}r\cos\theta \cdot r\sin \theta \sqrt{1+r^2}\cdot rd\theta dr$
$=\int_0^1\int_0^{\pi/2}r^2\sqrt{1+r^2}\sin\theta \cos \theta d\theta dr$
$=\int_0^1r^2\sqrt{1+r^2}\frac{\sin^2\theta}{2}]_0^{\pi/2}dr$
$=\int_0^1\frac{r^2\sqrt{1+r^2}}{2} dr$
$\approx 0.2101$
