The Great Gatsby, to set back

Could you please tell me the precise meaning of "set us all back" in the following excerpt from the chapter five of The Great Gatsby?

“We haven’t met for many years,” said Daisy, her voice as matter-of-fact as it could ever be.

“Five years next November.”

The automatic quality of Gatsby’s answer set us all back at least another minute. I had them both on their feet with the desperate suggestion that they help me make tea in the kitchen when the demoniac Finn brought it in on a tray.

Thank you.