Do you think its fair that one person in the community(the receiver) should have to be able to burdened and pained by memories so that no one else is?

Of course this isn't fair on many levels. The stress on the Giver/receiver is enormous. It is also cheating the people in the community out of feeling the passions that make life worth living. Society ends up being largely vacuous with one person that lives isolation and feels nothing but stress.