Answer
$\displaystyle \frac{4\sqrt{2}-2}{15}$
Work Step by Step
$\displaystyle \int_{0}^{1}\int_{0}^{1}xy\sqrt{x^{2}+y^{2}}dydx=\int_{0}^{1}[\int_{0}^{1}xy\sqrt{x^{2}+y^{2}}dy]dx=\left[\begin{array}{ll}
t=x^{2}+y^{2} & \\
dt=2ydy & ydy=dt/2\\
y=0\Rightarrow t=x^{2}, & y=1\Rightarrow t=x^{2}+1
\end{array}\right]$
... treat $x$ as a constant in the inner integral.
$=\displaystyle \int_{0}^{1}[\frac{x}{2}\int_{x^{2}}^{x^{2}+1}t^{1/2}dt]dx$
$=\displaystyle \int_{0}^{1}\frac{x}{2}\cdot[\frac{t^{3/2}}{3/2}]_{x^{2}}^{x^{2}+1}dx$
$=\displaystyle \frac{1}{3}\int_{0}^{1}x[(x^{2}+1)^{3/2}-x^{3}]dx$
$=\displaystyle \frac{1}{3}\int_{0}^{1}[x(x^{2}+1)^{3/2}-x^{4}]dx=\frac{1}{3}\{\int_{0}^{1}x(x^{2}+1)^{3/2}dx-\int_{0}^{1}x^{4}dx\}$
... for the first integral, $\left[\begin{array}{ll}
t=x^{2}+1 & \\
dt=2xdx & \\
x=0\Rightarrow t=1, & x=1\Rightarrow t=2
\end{array}\right]$
$=\displaystyle \frac{1}{3}\{\int_{1}^{2}\frac{1}{2}t^{3/2}dt-[\frac{x^{5}}{5}]_{0}^{1}\}$
$=\displaystyle \frac{1}{3}\{[\frac{t^{5/2}}{2\cdot(5/2)}]_{1}^{2}-[\frac{1-0}{5}] \}$
$=\displaystyle \frac{1}{15}[2^{5/2}-1-1]$
$=\displaystyle \frac{2}{15}(2\sqrt{2}-1)$
$=\displaystyle \frac{4\sqrt{2}-2}{15}$