Answer
$\dfrac{713}{180}$
Work Step by Step
Since, $\iint_S F \cdot dS=\iint_D [P \dfrac{\partial h}{\partial x}-Q+R\dfrac{\partial h}{\partial z}]dA $
where, $D$ is projection of $S$ onto xz-plane.
$\iint_S F \cdot n dS=\iint_D -xy (-2x) -(yz) (-2y) +(zx) dA$
$= \int_{0}^{1} \int_0^{1} 2x^2y+2y^2z+zx dy dx$
$= \int_{0}^{1} \int_0^{1} 2x^2y+2y^2(4-x^2-y^2)+(4-x^2-y^2)x dy dx$
$= \int_{0}^{1} x^2+(8/3) -(2x^2/3)-(2/5) +4x-x^3-\dfrac{x}{3} dx$
or, $=[\dfrac{34x}{15}+\dfrac{x^3}{9}+\dfrac{11x^2}{6}-\dfrac{x^4}{4}]_{0}^{1}$
Hence, $Flux=\dfrac{713}{180}$
