Answer
$a=0.1m$ and $b=0.3m$
Work Step by Step
1) Case a):
The lever arm between $F_1$ and the axis of rotation is $b$, and $F_1$ causes the square to rotate counterclockwise, so $\tau_1\gt0$ . Therefore, $\tau_1=+F_1b$
The lever arm between $F_2$ and the axis of rotation is $a$, and $F_2$ causes the square to rotate clockwise, so $\tau_2\lt0$ . Therefore, $\tau_2=-F_2a$
The net torque is zero, so $$F_1b-F_2a=0$$ $$F_1b=F_2a$$ $$\frac{b}{a}=\frac{F_2}{F_1}=3 (1)$$
2) Case b):
In case b), remember that the side of the square is $1m$.
The lever arm between $F_1$ and the axis of rotation is $1-a$, and $F_1$ causes the square to rotate counterclockwise, so $\tau_1\gt0$ . Therefore, $\tau_1=+F_1(1-a)$
The lever arm between $F_2$ and the axis of rotation is $b$, and $F_2$ causes the square to rotate clockwise, so $\tau_2\lt0$ . Therefore, $\tau_2=-F_2b$
The net torque is zero, so $$F_1(1-a)-F_2b=0$$ $$F_1(1-a)=F_2b$$ $$\frac{1-a}{b}=\frac{F_2}{F_1}=3 (2)$$
Solving (1) and (2), we get $a=0.1m$ and $b=0.3m$