Answer
after 3 "remove-top up" procedures:
$2.56$ gal.
after 5 "remove-top up" procedures:
$1.6384$ gal.
after n "remove-top up" procedures:
$5(\displaystyle \frac{3}{4})^{n}$
Work Step by Step
With $a_{n}$, we denote the amount of water that remains after a certain number of repetitions of the process.
1.
Starting with 5 gallons of water, $a_{1}=a=5$,
2.
Removing 1 gallon means $\displaystyle \frac{1}{5}$ of the water is removed,
so $\displaystyle \frac{4}{5}$ of $a_{1}$ remains,
$a_{2}=\displaystyle \frac{4}{5}a_{1}=(\frac{4}{5})(5)$,
Topping up with antifreeze, we have 5 galons of mixture.
3. Removing 1 gallon of the mixture ( $\displaystyle \frac{1}{5}$ of the mixture)
means removing $\displaystyle \frac{1}{5}$ of the water,
meaning that $\displaystyle \frac{4}{5}$ of $a_{2}$ remains,
$a_{3}=\displaystyle \frac{4}{5}a_{2}=(\frac{4}{5})^{2}(5)$
Topping up with antifreeze, we have 5 gallons of mixture.
4. Removing 1 gallon of the mixture ( $\displaystyle \frac{1}{5}$ of the mixture)
means removing $\displaystyle \frac{1}{5}$ of the water,
meaning that $\displaystyle \frac{4}{5}$ of $a_{3}$ remains,
$a_{4}=\displaystyle \frac{4}{5}a_{3}=(\frac{4}{5})(5)$
...
This pattern defines
$a_{n}=5(\displaystyle \frac{3}{4})^{n-1}$
as the amount of water
after (n-1) "remove-top up" procedures.
after 3 "remove-top up" procedures:
$a_{4}=(\displaystyle \frac{3}{4})^{3}(5)=\frac{64}{25}=2.56$ gal.
after 5 "remove-top up" procedures:
$a_{6}=(\displaystyle \frac{3}{4})^{5}(5)=\frac{1024}{625}=1.6384$ gal.
after n "remove-top up" procedures:
$a_{n+1}=5(\displaystyle \frac{3}{4})^{n}$