Answer
$\dfrac{4(-2+4\sqrt {2}+9\sqrt 3)}{105}$
Work Step by Step
The flux through a surface can be defined only when the surface is orientable.
We know that $\iint_S F \cdot dS=\iint_S F \cdot n dS$
Here, $n$ denotes the unit vector.
Since, $\iint_S y dS =\iiint_{D}y \sqrt {(x^{1/2})^2+(\sqrt y)^2+1^2} dA= \iint_D y\sqrt {x+y+1} dA$
$=\int_{0}^{1} \int_{0}^{1}y\sqrt {x+y+1} dy dx$
$=(2/3) \int_{0}^{1}y(y+2)^{3/2}-y(y+1)^{3/2} dy$
$=(2/3) \int_{0}^{1}y(y+2)^{3/2} dy -(2/3) \int_0^1 y(y+1)^{3/2} dy$
Plug $y=a^2-2$ in the first integral and $y=b^2-1$ in the second integral.
$=(2/3) \int_{\sqrt 2}^{\sqrt 3} (a^2-2) a^3 (2a) da -(2/3) \int_{1}^{\sqrt 2} (b^2-1) b^3 (2b) db$
$=(4/3) \int_{\sqrt 2}^{\sqrt 3} a^6-2a^4 dt- (4/3) \int_{1}^{\sqrt 2} (b^6-b^4) db$
$=\dfrac{4(-2+4\sqrt {2}+9\sqrt 3)}{105}$
