Why were the neighbors suspicious of Mr. and Mrs. Radley?

Mrs. Radley isn't mentioned in the first chapter. Mr. Radley is described and mean and unfriendly.

Mr. Radley was not a nice man. He went to town each day but never spoke to us even if we said “Good Morning, Sir.”

When he died, Calpurnia said, “There goes the meanest man God ever blew breath into.” The neighborhood thought maybe Boo would come out, but his older brother Nathan moved in and he was just as mean.