Answer
$\begin{aligned} \int_{0}^{a} \int_{0}^{\sqrt{a^{2}-x^{2}}} \frac{d y d x}{\left(x^{2}+y^{2}+1\right)^{\frac{3}{2}}} =\left[1-\frac{1}{\sqrt{a^{2}+1}}\right]\frac{\pi}{2} \end{aligned}$
Work Step by Step
The $y$ integration runs from $0=y$ to $\sqrt{a^{2}-x^{2}}=y$.
From the x-integration, we see that $x$ varies from 0 to $a$; thus, we conclude that the region of integration is in the first quadrant.
$\begin{aligned} \int_{0}^{a} \int_{0}^{\sqrt{a^{2}-x^{2}}} \frac{d y d x}{\left(x^{2}+y^{2}+1\right)^{\frac{3}{2}}} &=\int_{R} \int \frac{1}{\left(x^{2}+y^{2}+1\right)^{\frac{3}{2}}} d A \\ &=\int_{0}^{\frac{\pi}{2}} \int_{0}^{a} \frac{r d r d \theta}{\left(1+r^{2}\right)^{\frac{3}{2}}} \\ &=\left.\frac{1}{2} \int_{0}^{\frac{\pi}{2}}\left[\frac{2}{\sqrt{1+r^{2}}}\right]\right|_{0} ^{a} d \theta \\ &=\int_{0}^{\frac{\pi}{2}}\left[\frac{1}{\sqrt{1+a^{2}}}-1\right] d \theta \\ &=\left.\left[\frac{\theta}{\sqrt{1+a^{2}}}-\theta\right]\right|_{0} ^{\frac{\pi}{2}} \\ &=\frac{\pi}{2}\left[1-\frac{1}{\sqrt{1+a^{2}}}\right] \end{aligned}$
