Answer
The mass of the lamina is
$$\frac{1}{3} a b\left(3+a^{2}+b^{2}\right) $$
and the center of mass is at the point
$$
(\overline{x} ,\overline{y} )=\left(\frac{a\left(6+3 a^{2}+2 b^{2}\right)}{4\left(3+a^{2}+b^{2}\right)}, \frac{b\left(6+2 a^{2}+3 b^{2}\right)}{4\left(3+a^{2}+b^{2}\right)}\right)
$$.
Work Step by Step
$$
D=\left\{(x, y) | 0 \leq x \leq a , \quad 0 \leq y \leq b \right\}.
$$
The mass of the lamina is given by
$$
\begin{aligned} m &=\iint_{D} \rho(x, y) d A \\
&=\int_{0}^{a} \int_{0}^{b}\left(1+x^{2}+y^{2}\right) d y d x \\
&=\int_{0}^{a}\left[y+x^{2} y+\frac{1}{3} y^{3}\right]_{y=0}^{y=b} d x \\
& =\int_{0}^{a}\left(b+b x^{2}+\frac{1}{3} b^{3}\right) d x \\
&=\left[b x+\frac{1}{3} b x^{3}+\frac{1}{3} b^{3} x\right]_{0}^{a} \\
&=a b+\frac{1}{3} a^{3} b+\frac{1}{3} a b^{3} \\
&=\frac{1}{3} a b\left(3+a^{2}+b^{2}\right)
\end{aligned}
$$
The center of mass of the lamina is given by
$$
\begin{aligned} M_{y} &=\iint_{D} x \rho(x, y) d A\\
&=\int_{0}^{a} \int_{0}^{b}\left(x+x^{3}+x y^{2}\right) d y d x \\
&=\int_{0}^{a}\left[x y+x^{3} y+\frac{1}{3} x y^{3}\right]_{y=0}^{y=b} d x \\
&=\int_{0}^{a}\left(b x+b x^{3}+\frac{1}{3} b^{3} x\right) d x \\ &=\left[\frac{1}{2} b x^{2}+\frac{1}{4} b x^{4}+\frac{1}{6} b^{3} x^{2}\right]_{0}^{a}\\
&=\frac{1}{2} a^{2} b+\frac{1}{4} a^{4} b+\frac{1}{6} a^{2} b^{3} \\
&=\frac{1}{12} a^{2} b\left(6+3 a^{2}+2 b^{2}\right),
\end{aligned}
$$
and
$$
\begin{aligned} M_{x} &=\iint_{D} y \rho(x, y) d A \\
&=\int_{0}^{a} \int_{0}^{b}\left(y+x^{2} y+y^{3}\right) d y d x \\
& =\int_{0}^{a}\left[\frac{1}{2} y^{2}+\frac{1}{2} x^{2} y^{2}+\frac{1}{4} y^{4}\right]_{y=0}^{y=b} d x \\
& =\int_{0}^{a}\left(\frac{1}{2} b^{2}+\frac{1}{2} b^{2} x^{2}+\frac{1}{4} b^{4}\right) d x \\
&=\left[\frac{1}{2} b^{2} x+\frac{1}{6} b^{2} x^{3}+\frac{1}{4} b^{4} x\right]_{0}^{a} \\
& =\frac{1}{2} a b^{2}+\frac{1}{6} a^{3} b^{2}+\frac{1}{4} a b^{4} \\
&=\frac{1}{12} a b^{2}\left(6+2 a^{2}+3 b^{2}\right) \end{aligned}
$$
Hence, the center of mass is at the point
$$
\begin{aligned}(\overline{x}, \overline{y}) &=\left(\frac{M_{y}}{m}, \frac{M_{x}}{m}\right) \\
& =\left(\frac{\frac{1}{12} a^{2} b\left(6+3 a^{2}+2 b^{2}\right)}{\frac{1}{3} a b\left(3+a^{2}+b^{2}\right)}, \frac{\frac{1}{12} a b^{2}\left(6+2 a^{2}+3 b^{2}\right)}{\frac{1}{3} a b\left(3+a^{2}+b^{2}\right)}\right) \\ &=\left(\frac{a\left(6+3 a^{2}+2 b^{2}\right)}{4\left(3+a^{2}+b^{2}\right)}, \frac{b\left(6+2 a^{2}+3 b^{2}\right)}{4\left(3+a^{2}+b^{2}\right)}\right) \end{aligned}
$$
