Answer
$\dfrac{713}{180}$
Work Step by Step
The flux through a surface can be defined only when the surface is orientable.
We know that $\iint_S F \cdot dS=\iint_S F \cdot n dS$
Here, $n$ denotes the unit vector.
Since, $\iint_S F \cdot dS=\iint_D [P \dfrac{\partial h}{\partial x}-Q+R\dfrac{\partial h}{\partial z}]dA $
$\iint_S F \cdot n dS=\iint_D -xy (-2x) -(yz) (-2y) +(zx) dA= \int_{0}^{1} \int_0^{1} [2x^2y+2y^2z+zx] dy dx$
or, $= \int_{0}^{1} \int_0^{1} [2x^2y+2y^2(4-x^2-y^2)+(4-x^2-y^2) x] dy dx$
or, $= \int_{0}^{1} [x^2+\dfrac{8}{3} -\dfrac{2x^2}{3}-\dfrac{2}{5} +4x-x^3-\dfrac{x}{3}] dx$
Hence, the flux is: $\iint_S F \cdot n dS=[\dfrac{34x}{15}+\dfrac{x^3}{9}+\dfrac{11x^2}{6}-\dfrac{x^4}{4}]_{0}^{1}=\dfrac{713}{180}$
