Answer
$\dfrac{ \pi}{12}(8-5\sqrt 2)$
Work Step by Step
Since, $\iint_S F \cdot dS=\iint_D [P \dfrac{\partial h}{\partial x}-Q+R\dfrac{\partial h}{\partial z}]dA $
where, $D$ is projection of $S$ onto xz-plane.
$\iint_S F \cdot n dS=\iint_D \dfrac{y^2}{\sqrt {1-x^2-y^2}} dx dy= \int_{0}^{2 \pi} \int_0^{(1/\sqrt 2)} \dfrac{r^2}{ \sqrt {1-r^2}} \times \sin^2 \theta \times (r dr d\theta)$
or, $\iint_S F \cdot n dS=\int_{0}^{ 2\pi} \sin^2 \theta d\theta \times \int_0^{1/\sqrt 2} \dfrac{r^3}{ \sqrt {1-r^2}} dr$
Suppose$ 1-r^2=p; dp=-2r dr$
$\iint_S F \cdot n dS=\int_{0}^{ 2\pi} 0.5-0.5 \cos (2 \theta) d\theta \times \int_0^{1/\sqrt 2} \dfrac{1-p}{ \sqrt p} (dp/-2)$
Hence, we have $\iint_S F \cdot n dS=\pi [\dfrac{p\sqrt p}{3}- p^{1/2}]_1^{1/2}=\dfrac{ \pi}{12}(8-5\sqrt 2)$
