why does macbeth say that there is no one he fears but banquo

Banquo is the only person, besides L. Macbeth, that knows about he witches. Also, the witches pronounced Banquo's sons to be kings leaving Macbeth with a "fruitless crown" (no sons being kings).