Answer
$\dfrac{713}{180}$
Work Step by Step
Since, we have $\iint_S F \cdot dS=\iint_D [P \dfrac{\partial h}{\partial x}-Q+R\dfrac{\partial h}{\partial z}]dA $
where, $D$ is projection of $S$ onto xz-plane.
$\iint_S F \cdot n dS=\iint_D -xy (-2x) -(yz) (-2y) +(zx) dA= \int_{0}^{1} \int_0^{1} [2x^2y+2y^2z+zx] dy dx$
and $\iint_S F \cdot n dS= \int_{0}^{1} \int_0^{1} 2x^2y+2y^2 \times (4-x^2-y^2)+(4-x^2-y^2) \times (x) dy dx$
or, $\iint_S F \cdot n dS= \int_{0}^{1} x^2+(\dfrac{8}{3}) -(\dfrac{2x^2}{3})-(\dfrac{2}{5}) +4x-x^3-\dfrac{x}{3} dx$
or, $\iint_S F \cdot n dS=[\dfrac{34x}{15}+\dfrac{x^3}{9}+\dfrac{11x^2}{6}-\dfrac{x^4}{4}]_{0}^{1}=\dfrac{713}{180}$
