Answer
The area of the largest triangle is $~~2ab$
Work Step by Step
Let $(x,y)$ be the point where the corner of the rectangle intersects the ellipse in the first quadrant.
We can find an expression for $y$:
$\frac{y^2}{b^2} = 1 - \frac{x^2}{a^2}$
$y^2 = \frac{b^2}{a^2}~(a^2 -x^2)$
$y = \frac{b}{a}~\sqrt{a^2 -x^2}$
We can write an expression for the area of the rectangle:
$A = (2x)(2y)$
$A = 4xy$
$A = (4x)(\frac{b}{a}~\sqrt{a^2 -x^2})$
We can find $\frac{dA}{dx}$:
$\frac{dA}{dx} = \frac{4b}{a}~\sqrt{a^2 -x^2}+(\frac{4xb}{a})(\frac{-x}{\sqrt{a^2-x^2}})$
$\frac{dA}{dx} = \frac{(4b)(a^2-x^2)}{a~\sqrt{a^2-x^2}} -\frac{4x^2b}{a~\sqrt{a^2-x^2}}$
$\frac{dA}{dx} = \frac{4a^2b-4x^2b}{a~\sqrt{a^2-x^2}} -\frac{4x^2b}{a~\sqrt{a^2-x^2}}$
$\frac{dA}{dx} = \frac{4a^2b-8x^2b}{a~\sqrt{a^2-x^2}} = 0$
$4a^2b-8x^2b = 0$
$8x^2b = 4a^2b$
$x^2 = \frac{a^2}{2}$
$x = \frac{a}{\sqrt{2}}$
We can find $y$:
$y = \frac{b}{a}~\sqrt{a^2 -x^2}$
$y = \frac{b}{a}~\sqrt{a^2 -(\frac{a}{\sqrt{2}})^2}$
$y = \frac{b}{a}~\sqrt{\frac{a^2}{2}}$
$y = \frac{b}{\sqrt{2}}$
We can find the area of the rectangle:
$A = 4xy$
$A = 4(\frac{a}{\sqrt{2}})(\frac{b}{\sqrt{2}})$
$A = 2ab$
The area of the largest triangle is $~~2ab$