Answer
$4 \pi R^5$
Work Step by Step
The Divergence Theorem states that $\iint_S \overrightarrow{F}\cdot d\overrightarrow{S}=\iiint_Ediv \overrightarrow{F}dV $
Here, $S$ is a closed surface and $E$ is the region inside that surface.
$div F=\dfrac{\partial a}{\partial x}+\dfrac{\partial b}{\partial y}+\dfrac{\partial c}{\partial z}$
This implies that $div F=(x^2+y^2+z^2) \cdot \lt x,y,z \gt=3(x^2+y^2+z^2)+2(x^2+y^2+z^2)=5(x^2+y^2+z^2)=5\rho^2 $
Now, we have
$Flux=\int_{0}^{2 \pi}\int_0^{\pi} \int_{0}^{R} 5\rho^2 dv$
$=\int_{0}^{2 \pi}\int_0^{\pi} \int_{0}^{R} 5\rho^2 \sin \phi d \rho d\phi d \theta$
$=\int_{0}^{2 \pi}\int_0^{\pi} \int_{0}^{R} 5\rho^2 \sin \phi d \rho d\phi d \theta$
$=\int_{0}^{2 \pi}\int_0^{\pi} R^5 \sin \phi d \rho d\phi d \theta$
$=\int_{0}^{2 \pi} R^5 [(-\cos \phi)_0^{\pi} d \theta$
$=\int_{0}^{2 \pi} 2 R^5 d \theta$
$=4 \pi R^5$
