According to the giver , what is challenging about having a family?

The Giver tells him that being the Receiver makes family life difficult. The Receiver has so much emotional strain and is so much more aware of complex emotion where as the his family remains superficial.