Answer
We should use the #18 wire.
Work Step by Step
Recalling that the magnetic field strength inside a solenoid is given by
$$B=\dfrac{\mu_0NI}{l}$$
The author told us that we should wound the coils as close together as possible. This means that the length of the coil is given by $l=ND$ where $D$ is the diameter of the wire. So, $N=l/D$
$$B=\dfrac{\mu_0 \color{red}{\bf\not} lI}{\color{red}{\bf\not} lD}$$
$$B=\dfrac{\mu_0I}{D}$$
Hence, the current needed is given by
$$I=\dfrac{DB}{\mu_0}$$
For wire #18: which has a maximum current of 5 A, the current needed to create a magnetic field of 5 mT is;
$$I_{\rm \#18}=\dfrac{D_{\#18}B}{\mu_0}=\dfrac{(1.02\times 10^{-3})(5\times 10^{-3})}{4\pi \times 10^{-7}}=\bf 4.06\;\rm A$$
This current is in the limit, so we can use this wire.
For wire #26: which has a maximum current of 1 A, the current needed to create a magnetic field of 5 mT is;
$$I_{\rm \#26}=\dfrac{D_{\#26}B}{\mu_0}=\dfrac{(0.41\times 10^{-3})(5\times 10^{-3})}{4\pi \times 10^{-7}}=\bf 1.63\;\rm A$$
This current exceeds the limit, so we can't use this wire.
