Why is Atticus worried about Tom Robinson's trial?

Atticus says that the circumstances of the case couldn't worse - the case comes down to a black man's word against the word of the white Ewell family, and the jury couldn't possibly take Tom's word over the word of white people.