Why does Jem conclude that Boo Radley stays inside?

From the text:

other? Scout, I think I’m beginning to understand something. I think I’m beginning to understand why Boo Radley’s stayed shut up in the house all this time... it’s because he wants to stay inside.”