Which character is mostly responsible for antagonizing Boo Radley? How is this?

Although it would seem that Scout, Jem and Dill antagonize Boo Radley, we find out that he quite enjoyed their antics. His greater antagonists were his father and brother who kept him locked and isolated in his house during the day.