Answer
The net magnetic force on wire 5 due to the currents in the other wires is $~~(-4.69\times 10^{-4}~N)~\hat{j}$
Work Step by Step
We can write the general expression for the magnetic force on a current-carrying wire due to a parallel current-carrying wire:
$F = \frac{\mu_0~L~i_1~i_2}{2\pi~d}$
Note that currents in the same direction attract each other.
The force on wire 5 due to the other wires is directed in the -y direction.
We can find the magnitude of the magnetic force on wire 5 due to the other wires:
$F = \frac{\mu_0~L~i^2}{2\pi~d}+\frac{\mu_0~L~i^2}{(2\pi)~(2d)}+\frac{\mu_0~L~i^2}{(2\pi)~(3d)}+\frac{\mu_0~L~i^2}{(2\pi)~(4d)}$
$F = \frac{12~\mu_0~L~i^2}{2\pi~(12d)}+\frac{6~\mu_0~L~i^2}{(2\pi)~(12d)}+\frac{4~\mu_0~L~i^2}{(2\pi)~(12d)}+\frac{3~\mu_0~L~i^2}{(2\pi)~(12d)}$
$F = \frac{25~\mu_0~L~i^2}{(2\pi)~(12d)}$
$F = \frac{(25)(4\pi\times 10^{-7}~H/m)(10.0~m)(3.00~A)^2}{(2\pi)(12)(0.0800~m)}$
$F = 4.69\times 10^{-4}~N$
The net magnetic force on wire 5 due to the currents in the other wires is $~~(-4.69\times 10^{-4}~N)~\hat{j}$