Answer
$(0,0)$, $(0,1)$, and $(1,0)$ are saddle points and $\left(\frac{1}{3}, \frac{1}{3}\right)$ is a local maximum point.
Work Step by Step
Let us first find the critical points of the function where $f_x$ and $f_y$ are equal to 0, assuming first and second partial derivatives are continuous.
$f_x=y-2xy-y^2=y(1-2x-y)$ and $f_y=x-x^2-2xy=x(1-x-2y)$
We can see that $(0,0)$ is a critical point.
Two other critical points are $\left(0, 1\right)$ and $\left(1, 0\right)$.
To find the other critical point, let us substitute $y=1-2x$ into the second equation.
$x(1-x-2(1-2x))=x(3x-1)$, which is equal to 0 when $x=\frac{1}{3}$ and $y=\frac{1}{3}$.
The two critical points are $(0,0)$ and $\left(\frac{1}{3}, \frac{1}{3}\right)$.
$f_{xx}=-2y$ and $f_{yy}=-2x$ and $f_{xy}=1-2x-2y$.
$D(0,0)=(0)(0)-(1)^2<0$, which means $(0,0)$ is a saddle point.
$D(0,1)=(-2)(0)-(-1)^2<0$, which means $(0,1)$ is a saddle point.
$D(1,0)=(0)(-2)-(-1)^2<0$, which means $(1,0)$ is a saddle point.
$D\left(\frac{1}{3}, \frac{1}{3}\right)=\left(\frac{-2}{3}\right)\left(\frac{-2}{3}\right)-\left(\frac{-1}{3}\right)^2>0$ and $f_{xx}<0$, which means $\left(\frac{1}{3}, \frac{1}{3}\right)$ is a local maximum point.
$f\left(\dfrac{1}{3},\dfrac{1}{3}\right)=\dfrac{1}{27}$
