Answer
(a) The maximum force is $100N$
(b) The maximum force is $29.4N$
Work Step by Step
(a) When the force $P$ is applied to the more massive block $(M=12kg)$, it is in fact applied to the system of both blocks, including the less massive one $(m=5kg)$ as this system moves with acceleration $a$.
$$P=(M+m)a$$
We do not have $a$ for now, so we need to find $a$ by examining the forces on the smaller block.
Block $m$ moves forward with acceleration $a$ along with the system, which means block $M$ exerts force $p$ on block $m$ as $M$ is brought to motion by force $P$. From Newton's 2nd Law: $$p=ma$$ This $p$ force, however, creates a reaction force $-p$ which propels block $m$ to move backward, but it is opposed by static friction $f_s$. The maximum $p$ can get before surpassing friction is $$p_{max}=f_s^{max}=\mu_sF_N=\mu_smg$$
Combining 2 equations above, we have $$ma=\mu_smg$$ $$a_{max}=\mu_sg=0.6\times9.8=5.88m/s^2$$
This is the maximum acceleration block $m$ can get without getting it to slide. From here, we find $P_{max}$ $$P_{max}=(M+m)a_{max}=100N$$
(b) When force $P$ is applied directly to block $m$, it is also directly opposed by static friction $f_s$. There is no more system, and we care only about the smaller block.
$$P_{max}=f_s^{max}=\mu_smg=0.6\times5\times9.8=29.4N$$
