Answer
$\dfrac{ \pi}{12}(8-5\sqrt 2)$
Work Step by Step
Since, $\iint_S F \cdot dS=\iint_D [P \dfrac{\partial h}{\partial x}-Q+R\dfrac{\partial h}{\partial z}]dA $
where, $D$ is projection of $S$ onto xz-plane.
$\iint_S F \cdot n dS=\iint_D \dfrac{y^2}{\sqrt {1-x^2-y^2}} dx dy$
$= \int_{0}^{2 \pi} \int_0^{1/\sqrt 2} \dfrac{r^2 \sin^2 \theta}{ \sqrt {1-r^2}}(r dr d\theta)$
$=\int_{0}^{ 2\pi} \sin^2 \theta d\theta \times \int_0^{1/\sqrt 2} \dfrac{r^3}{ \sqrt {1-r^2}} dr$
Plug $a=1-r^2 \implies da=-2r dr$
$=\int_{0}^{ 2\pi} (1/2)-(1/2) \cos (2 \theta) d\theta \times \int_0^{1/\sqrt 2} \dfrac{1-a}{ \sqrt a} (da/-2)$
$=\pi [\dfrac{a\sqrt a}{3}- \sqrt a]_1^{1/2}$
$=\dfrac{ \pi}{12}(8-5\sqrt 2)$
