Apparently, Atticus has known all along about the night Jem lost his pants on the Radleys’ fence, but he never felt the need to say anything about it. What physical evidence led Atticus to make his realization?

First, Atticus is by no means stupid..... his son came home without his pants, and even if he'd been playing strip poker, he'd have his pants when he got home. Second, when his pants appear, they're mended...... poorly mended, meaning Calpurnia didn't fix them.