Why was Griffin unable to eat at the restaurant?

From the text:

"I went into a place and was already ordering lunch, when it occurred to me that I could not eat unless I exposed my invisible face. I finished ordering the lunch, told the man I should be back in ten minutes, and went out exasperated. I don’t know if you have ever been disappointed in your appetite.”