What had Hally failed to realize, as a boy, about the park bench Sam left him on?

Hally failed to realize that Sam had left him sitting on a "white's only" bench.

It was a "Whites Only" bench. You were too young, too excited to notice then. But not anymore. If you're not careful . . . Master Harold . . . you're going to be sitting up there by yourself for a long time to come, and there won't be a kite in the sky.