Johnny said that the Lytes would never come back to the house. Why did he say that?

Johnny tells Cilla that the Revolution will prevent the Lytes from ever coming back here. The Lyte's had ties to the British or the Whigs because there was money to be made. Too many people would be after him if he came back.