Answer
$\dfrac{4 \sqrt 2}{15}-\dfrac{1}{3} $
Work Step by Step
The spherical coordinates system can be expressed as:
$x=\rho \sin \phi \cos \theta \\ y=\rho \sin \phi \sin \theta $ and $z=\rho \cos \phi$
So, $\rho=\sqrt {x^2+y^2+z^2} \implies \rho^2=x^2+y^2+z^2$
The jacobian for spherical coordinates can be written as:
$\phi^2 \sin \phi$.
Now,
$\int_{0}^1 \int_{0}^{\sqrt {1-x^2}} \int_{\sqrt {x^2+y^2}}^{\sqrt {2-x^2-y^2}} x y dz dy dx =\int_0^{\pi/4} \int_0^{\pi/2} \int_0^{\sqrt 2} (\rho \sin \phi \cos \theta) (\rho \sin \phi \sin \theta) \rho^2 \sin \phi d\rho d \theta d \phi $
or, $=\int_0^{\sqrt 2} \rho^4 d \rho \int_0^{\pi/2} \sin \theta \cos \theta d \theta \int_0^{\pi/4} \sin^3 \phi d \phi$
or, $=\dfrac{4\sqrt 2}{5}\times \dfrac{1}{2} \times (\dfrac{\sqrt 2}{12}-\dfrac{\sqrt 2}{2}-\dfrac{1}{3}+1) $
or, $=\dfrac{4 \sqrt 2}{15}-\dfrac{1}{3} $
