Answer
Equilateral triangle with side $r\sqrt 3$
Work Step by Step
Using the notations from the picture, the area of the isosceles triangle is:
$A(x)=\frac{(2\sqrt{r^2-x^2})(r+x)}{2}=\sqrt{r^2-x^2}(r+x),$
where $2\sqrt{r^2-x^2}$ is the base and $r+x$ is the height.
In order to find the maximum of $A$, we calculate the derivative:
$$\begin{aligned}
A'(x)&=\frac{-2x}{2\sqrt{r^2-x^2}}(r+x)+\sqrt{r^2-x^2}\\
&=\frac{-x(r+x)}{\sqrt{r^2-x^2}}+\sqrt{r^2-x^2}\\
&=\frac{-rx-x^2+r^2-x^2}{\sqrt{r^2-x^2}}\\
&=\frac{-2x^2-rx+r^2}{\sqrt{r^2-x^2}}
\end{aligned}$$
Solve $A'(x)=0$:
$$\begin{aligned}
-2x^2-rx+r^2&=0\\
x&=\frac{r\pm\sqrt{9r^2}}{-2r}\\
&=\frac{r\pm 3r}{-2r}\\
x_1&=-r\\
x_2&=\frac{r}{2}.
\end{aligned}$$
Because $x\geq 0$, the only solution is $x==\frac{r}{2}$.
This means that the area reaches its maximum when the triangle is equilateral. Its side is:
$2\sqrt{r^2-\left(\frac{r}{2}\right)^2}=2\cdot\frac{r\sqrt 3}{2}=r\sqrt 3.$
