Why is Charlie so angry with himself as well as the crowd?

Charlie is angry with the cross and with himaelf because they we're making fun of the boy and it reminds him that at one point he was also mental y retarded Like that boy