Answer
We show that
$Q = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_S^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = \left\{ {\begin{array}{*{20}{c}}
{ - 4\pi Gm,}&{a < 0 < b,}&{m{\rm{ {\ }lies{\ } inside{\ } the{\ } cylinder}}}\\
{0,}&{0 < a < b,}&{m{\rm{ {\ }lies{\ } outside{\ } the{\ } cylinder}}}
\end{array}} \right.$
Work Step by Step
The flux through the cylinder is the sum of the flux through the bottom, through the top, and through the side. That is,
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_S^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} + \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} + \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{top}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}}$
Since ${{\bf{e}}_r} = \left( {\dfrac{x}{r},\dfrac{y}{r},\dfrac{z}{r}} \right)$, so ${\bf{F}} = - Gm\dfrac{{{{\bf{e}}_r}}}{{{r^2}}} = - Gm\left( {\dfrac{x}{{{r^3}}},\dfrac{y}{{{r^3}}},\dfrac{z}{{{r^3}}}} \right)$.
In cylindrical coordinates: ${\bf{F}} = - Gm\left( {\dfrac{{\cos \theta }}{{{r^2}}},\dfrac{{\sin \theta }}{{{r^2}}},\dfrac{z}{{{r^3}}}} \right)$.
Case 1. $a < 0 < b$ ($m$ lies inside the cylinder)
(1.1) through the bottom
The surface is the disk ${x^2} + {y^2} = {R^2}$ at $z=a$. In cylindrical coordinates, it can be parametrized by
$G\left( {\rho ,\theta } \right) = \left( {\rho \cos \theta ,\rho \sin \theta ,a} \right)$
${{\bf{T}}_\rho } = \left( {\cos \theta ,\sin \theta ,0} \right)$
${{\bf{T}}_\theta } = \left( { - \rho \sin \theta ,\rho \cos \theta ,0} \right)$
${\bf{N}}\left( {\rho ,\theta } \right) = {{\bf{T}}_\rho } \times {{\bf{T}}_\theta } = \left| {\begin{array}{*{20}{c}}
{\bf{i}}&{\bf{j}}&{\bf{k}}\\
{\cos \theta }&{\sin \theta }&0\\
{ - \rho \sin \theta }&{\rho \cos \theta }&0
\end{array}} \right| = \rho {\bf{k}}$
However, the normal vector is inward-pointing into the cylinder. So, we need to have an outward-pointing normal vector by setting ${\bf{N}}\left( {\rho ,\theta } \right) = - \rho {\bf{k}}$.
Since $z=a$, and ${\rho ^2} + {z^2} = {r^2}$, so
${\bf{F}}\left( {G\left( {\rho ,\theta } \right)} \right)\cdot{\bf{N}}\left( {\rho ,\theta } \right) = - Gm\left( {\dfrac{{\rho \cos \theta }}{{{{\left( {{\rho ^2} + {a^2}} \right)}^{3/2}}}},\dfrac{{\rho \sin \theta }}{{{{\left( {{\rho ^2} + {a^2}} \right)}^{3/2}}}},\dfrac{a}{{{{\left( {{\rho ^2} + {a^2}} \right)}^{3/2}}}}} \right)\cdot\left( {0,0, - \rho } \right)$
${\bf{F}}\left( {G\left( {\rho ,\theta } \right)} \right)\cdot{\bf{N}}\left( {\rho ,\theta } \right) = Gm\dfrac{{a\rho }}{{{{\left( {{\rho ^2} + {a^2}} \right)}^{3/2}}}}$
We evaluate the flux through the bottom of the cylinder using Eq. (3):
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\left( {G\left( {\rho ,\theta } \right)} \right)\cdot{\bf{N}}\left( {\rho ,\theta } \right){\rm{d}}\rho {\rm{d}}\theta $
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = Gma\mathop \smallint \limits_{\theta = 0}^{2\pi } \mathop \smallint \limits_{\rho = 0}^R \dfrac{\rho }{{{{\left( {{\rho ^2} + {a^2}} \right)}^{3/2}}}}{\rm{d}}\rho {\rm{d}}\theta $
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = Gma\mathop \smallint \limits_{\theta = 0}^{2\pi } {\rm{d}}\theta \mathop \smallint \limits_{\rho = 0}^R \dfrac{\rho }{{{{\left( {{\rho ^2} + {a^2}} \right)}^{3/2}}}}{\rm{d}}\rho $
Since $\mathop \smallint \limits_{\rho = 0}^R \dfrac{\rho }{{{{\left( {{\rho ^2} + {a^2}} \right)}^{3/2}}}}{\rm{d}}\rho = - \dfrac{1}{{\sqrt {{\rho ^2} + {a^2}} }}$, so
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gma\left( {\dfrac{1}{{\sqrt {{\rho ^2} + {a^2}} }}} \right)|_0^R$
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gma\left( {\dfrac{1}{{\sqrt {{R^2} + {a^2}} }} - \dfrac{1}{{\sqrt {{a^2}} }}} \right)$
Since in this case $a < 0$, so
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gma\left( {\dfrac{1}{{\sqrt {{R^2} + {a^2}} }} + \dfrac{1}{a}} \right)$
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gm\left( {\dfrac{a}{{\sqrt {{R^2} + {a^2}} }} + 1} \right)$
(1.2) through the side
The surface of the side of the cylinder ${x^2} + {y^2} = {R^2}$ for $a \le z \le b$ can be parametrized by
$G\left( {\theta ,z} \right) = \left( {R\cos \theta ,R\sin \theta ,z} \right)$
${{\bf{T}}_\theta } = \left( { - R\sin \theta ,R\cos \theta ,0} \right)$
${{\bf{T}}_z} = \left( {0,0,1} \right)$
${\bf{N}}\left( {\theta ,z} \right) = {{\bf{T}}_\theta } \times {{\bf{T}}_z} = \left| {\begin{array}{*{20}{c}}
{\bf{i}}&{\bf{j}}&{\bf{k}}\\
{ - R\sin \theta }&{R\cos \theta }&0\\
0&0&1
\end{array}} \right| = R\cos \theta {\bf{i}} + R\sin \theta {\bf{j}}$
At the side of the cylinder ${R^2} + {z^2} = {r^2}$, so
${\bf{F}}\left( {G\left( {\theta ,z} \right)} \right)\cdot{\bf{N}}\left( {\theta ,z} \right)$
$ = - Gm\left( {\dfrac{{R\cos \theta }}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}},\dfrac{{R\sin \theta }}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}},\dfrac{z}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}}} \right)\cdot\left( {R\cos \theta ,R\sin \theta ,0} \right)$
$ = - Gm\left( {\dfrac{{R\cos \theta }}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}},\dfrac{{R\sin \theta }}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}},\dfrac{z}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}}} \right)\cdot\left( {R\cos \theta ,R\sin \theta ,0} \right)$
$ = - Gm\left( {\dfrac{{{R^2}{{\cos }^2}\theta }}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}} + \dfrac{{{R^2}{{\sin }^2}\theta }}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}}} \right)$
${\bf{F}}\left( {G\left( {\theta ,z} \right)} \right)\cdot{\bf{N}}\left( {\theta ,z} \right) = - Gm\dfrac{{{R^2}}}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}}$
We evaluate the flux through the side of the cylinder using Eq. (3):
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\left( {G\left( {\theta ,z} \right)} \right)\cdot{\bf{N}}\left( {\theta ,z} \right){\rm{d}}\theta {\rm{d}}z$
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - Gm{R^2}\mathop \smallint \limits_{\theta = 0}^{2\pi } \mathop \smallint \limits_{z = a}^b \dfrac{1}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}}{\rm{d}}\theta {\rm{d}}z$
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - Gm{R^2}\mathop \smallint \limits_{\theta = 0}^{2\pi } {\rm{d}}\theta \mathop \smallint \limits_{z = a}^b \dfrac{1}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}}{\rm{d}}z$
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gm{R^2}\mathop \smallint \limits_{z = a}^b \dfrac{1}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}}{\rm{d}}z$
Since $\mathop \smallint \limits_{}^{} \dfrac{1}{{{{\left( {{R^2} + {z^2}} \right)}^{3/2}}}}{\rm{d}}z = \dfrac{z}{{{R^2}\sqrt {{R^2} + {z^2}} }}$, so
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gm\left( {\dfrac{z}{{\sqrt {{R^2} + {z^2}} }}} \right)|_a^b$
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gm\left( {\dfrac{b}{{\sqrt {{R^2} + {b^2}} }} - \dfrac{a}{{\sqrt {{R^2} + {a^2}} }}} \right)$
(1.3) through the top
Similar to part (1.3), but here we have $z=b$. The outward-pointing normal vector is ${\bf{N}}\left( {\rho ,\theta } \right) = \rho {\bf{k}}$.
So,
${\bf{F}}\left( {G\left( {\rho ,\theta } \right)} \right)\cdot{\bf{N}}\left( {\rho ,\theta } \right) = - Gm\left( {\dfrac{{\rho \cos \theta }}{{{{\left( {{\rho ^2} + {b^2}} \right)}^{3/2}}}},\dfrac{{\rho \sin \theta }}{{{{\left( {{\rho ^2} + {b^2}} \right)}^{3/2}}}},\dfrac{b}{{{{\left( {{\rho ^2} + {b^2}} \right)}^{3/2}}}}} \right)\cdot\left( {0,0,\rho } \right)$
${\bf{F}}\left( {G\left( {\rho ,\theta } \right)} \right)\cdot{\bf{N}}\left( {\rho ,\theta } \right) = - Gm\dfrac{{b\rho }}{{{{\left( {{\rho ^2} + {b^2}} \right)}^{3/2}}}}$
We evaluate the flux through the top of the cylinder using Eq. (3):
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{top}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{top}}}^{} {\bf{F}}\left( {G\left( {\rho ,\theta } \right)} \right)\cdot{\bf{N}}\left( {\rho ,\theta } \right){\rm{d}}\rho {\rm{d}}\theta $
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{top}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - Gmb\mathop \smallint \limits_{\theta = 0}^{2\pi } \mathop \smallint \limits_{\rho = 0}^R \dfrac{\rho }{{{{\left( {{\rho ^2} + {b^2}} \right)}^{3/2}}}}{\rm{d}}\rho {\rm{d}}\theta $
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{top}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - Gmb\mathop \smallint \limits_{\theta = 0}^{2\pi } {\rm{d}}\theta \mathop \smallint \limits_{\rho = 0}^R \dfrac{\rho }{{{{\left( {{\rho ^2} + {b^2}} \right)}^{3/2}}}}{\rm{d}}\rho $
Since $\mathop \smallint \limits_{\rho = 0}^R \dfrac{\rho }{{{{\left( {{\rho ^2} + {b^2}} \right)}^{3/2}}}}{\rm{d}}\rho = - \dfrac{1}{{\sqrt {{\rho ^2} + {b^2}} }}$, so
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{top}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = 2\pi Gmb\left( {\dfrac{1}{{\sqrt {{\rho ^2} + {b^2}} }}} \right)|_0^R = 2\pi Gmb\left( {\dfrac{1}{{\sqrt {{R^2} + {b^2}} }} - \dfrac{1}{b}} \right)$
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{top}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = 2\pi Gm\left( {\dfrac{b}{{\sqrt {{R^2} + {b^2}} }} - 1} \right)$
Thus, the total flux through the cylinder for $a < 0 < b$ is
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_S^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} + \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} + \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{top}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}}$
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_S^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gm\left( {\dfrac{a}{{\sqrt {{R^2} + {a^2}} }} + 1} \right)$
$ - 2\pi Gm\left( {\dfrac{b}{{\sqrt {{R^2} + {b^2}} }} - \dfrac{a}{{\sqrt {{R^2} + {a^2}} }}} \right)$
$ + 2\pi Gm\left( {\dfrac{b}{{\sqrt {{R^2} + {b^2}} }} - 1} \right)$
Therefore,
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_S^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 4\pi Gm$
for $a < 0 < b$ ($m$ lies inside the cylinder).
Case 2: $0 < a < b$ ($m$ lies outside the cylinder)
(1.1) through the bottom
Similar to Case 1 above, but here $a > 0$, so the integral
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gma\left( {\dfrac{1}{{\sqrt {{R^2} + {a^2}} }} - \dfrac{1}{{\sqrt {{a^2}} }}} \right)$
becomes
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gma\left( {\dfrac{1}{{\sqrt {{R^2} + {a^2}} }} - \dfrac{1}{a}} \right)$
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gm\left( {\dfrac{a}{{\sqrt {{R^2} + {a^2}} }} - 1} \right)$
(1.2) through the side
Same with Case 1 above, we get
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gm\left( {\dfrac{b}{{\sqrt {{R^2} + {b^2}} }} - \dfrac{a}{{\sqrt {{R^2} + {a^2}} }}} \right)$
(1.3) through the top
Same with Case 1 above, we get
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{top}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = 2\pi Gm\left( {\dfrac{b}{{\sqrt {{R^2} + {b^2}} }} - 1} \right)$
Thus, the total flux through the cylinder for $0 < a < b$ is
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_S^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{bottom}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} + \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{side}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} + \mathop \smallint \limits_{}^{} \mathop \smallint \limits_{{S_{top}}}^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}}$
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_S^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = - 2\pi Gm\left( {\dfrac{a}{{\sqrt {{R^2} + {a^2}} }} - 1} \right)$
$ - 2\pi Gm\left( {\dfrac{b}{{\sqrt {{R^2} + {b^2}} }} - \dfrac{a}{{\sqrt {{R^2} + {a^2}} }}} \right)$
$ + 2\pi Gm\left( {\dfrac{b}{{\sqrt {{R^2} + {b^2}} }} - 1} \right)$
Therefore,
$\mathop \smallint \limits_{}^{} \mathop \smallint \limits_S^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = 0$
Finally, our conclusion is
$Q = \mathop \smallint \limits_{}^{} \mathop \smallint \limits_S^{} {\bf{F}}\cdot{\rm{d}}{\bf{S}} = \left\{ {\begin{array}{*{20}{c}}
{ - 4\pi Gm,}&{a < 0 < b,}&{m{\rm{ {\ }lies{\ } inside{\ } the{\ } cylinder}}}\\
{0,}&{0 < a < b,}&{m{\rm{ {\ }lies{\ } outside{\ } the{\ } cylinder}}}
\end{array}} \right.$
