Why did Holmes hurry out as soon as his visitors had left ? Chapter 4

Holmes is suspicious of the two men. As soon as Sir Henry and Dr. Mortimer leave, Holmes jumps to action. He and Watson follow the men, and notice another man following them from a cab. Watson notes that this man has a bushy black beard and a pair of piercing eyes. When the cab suddenly rushes off, Holmes attempts to hail his own cab, but fails. Cursing his bad luck, Holmes admires the culprit's cleverness, but is glad he was able to note the cab number before it left.