Answer
(a) $T = 11.4~N$
(b) $m = 2.57~kg$
Work Step by Step
(a) Let's consider the system of the 5.00-kg box. We can find the force of kinetic friction $F_f$ acting on the box.
$F_f = F_N~\mu_k$
$F_f = (mg-F~sin(\theta))~\mu_k$
$F_f = ((5.00~kg)(9.80~m/s^2)-(40.0~N)~sin(53.1^{\circ}))\times (0.30)$
$F_f = 5.10~N$
We can use a force equation to find the tension $T$.
$\sum F = ma$
$F_x - T - F_f = ma$
$T = F_x - F_f - ma$
$T = [40.0~N~cos(53.1^{\circ})]-(5.10~N)-(5.00~kg)(1.50~m/s^2)$
$T = 11.4~N$
(b) Let's consider the system of box A.
$\sum F = ma$
$T - mg~\mu_k = ma$
$ma+ mg~\mu_k = T$
$m = \frac{T}{a+g~\mu_k}$
$m = \frac{11.4~N}{(1.50~m/s^2)+(9.80~m/s^2)(0.30)}$
$m = 2.57~kg$