Answer
Mass of lamina: $ab(\frac{a^2+b^2+3}{3})$
Center of lamina: $(\frac{a(3a^2+2b^2+12)}{4(a^2+b^2+3)},\frac{b(2a^2+3b^2+12)}{4(a^2+b^2+3)})$
Work Step by Step
Find the mass of the lamina:
$m=\iint_D\rho(x,y)dA=\int_0^a\int_0^b1+x^2+y^2dydx=\int_0^ay+x^2y+\frac{y^3}{3}]_0^bdx=\int_0^ab+bx^2+\frac{b^3}{3}dx=[bx+\frac{bx^3}{3}+\frac{b^3x}{3}]_0^a=ab+\frac{a^3b+ab^3}{3}=ab(\frac{a^2+b^2+3}{3})$
Find the moment of the lamina about x-axis:
$M_x=\iint_Dy\rho(x,y)dA=\int_0^a\int_0^by+x^2y+y^3 dydx=\int_0^a[\frac{y^2}{2}+\frac{x^2y^2}{2}+\frac{y^4}{4}]_0^bdx=\int_0^a\frac{b^2}{2}+\frac{b^2x^2}{2}+\frac{b^4}{4}dx=[\frac{b^2x}{2}+\frac{b^2x^3}{6}+\frac{b^4x}{4}]_0^a=\frac{ab^2}{2}+\frac{a^3b^2}{6}+\frac{ab^4}{4}=ab^2(\frac{2a^2+3b^2+12}{12})$
Find the moment of the lamina about the y-axis:
$M_y=\iint_Dx\rho(x,y)dA=\int_0^a\int_0^bx+x^3+xy^2dydx=\int_0^axy+x^3y+\frac{xy^3}{3}]_0^bdx=\int_0^abx+bx^3+\frac{b^3x}{3}dx=[\frac{bx^2}{2}+\frac{bx^4}{4}+\frac{b^3x^2}{6}]_0^a=\frac{a^2b}{2}+\frac{a^4b}{4}+\frac{a^2b^3}{6}=a^2b(\frac{3a^2+2b^2+12}{12})$
Then,
$\bar{x}=\frac{My}{m}=\frac{a^2b(\frac{3a^2+2b^2+12}{12})}{ab(\frac{a^2+b^2+3}{3})}=\frac{a(3a^2+2b^2+12)}{4(a^2+b^2+3)}$
and
$\bar{y}=\frac{Mx}{m}=\frac{ab^2(\frac{2a^2+3b^2+12}{12})}{ab(\frac{a^2+b^2+3}{3})}=\frac{b(2a^2+3b^2+12)}{4(a^2+b^2+3)}$
So, the center of the lamina is $(\frac{a(3a^2+2b^2+12)}{4(a^2+b^2+3)},\frac{b(2a^2+3b^2+12)}{4(a^2+b^2+3)})$.
