What is the implication of Mr. Gilmer calling Tom a boy? Why is Dill the one who gets upset by these references?

This was a degrading term used on adult black men by whites. It smacks of the old slavery rhetoric of the American South. I think Dill realizes the offensiveness of this.