“Nothing really mattered that much, not the good things and not the bad ones. We were in the business of mutual amusement, and we were reasonably prosperous.”

In context, Quentin is alluding to the fact that though his life is comfortable and without any significant trouble... it is also lacking joy. He is somewhere in between.