out there with him

I'd like to know if "him" (out there with him) in the following excerpt from The Great Gatsby refers to the boat:

On the green Sound, stagnant in the heat, one small sail crawled slowly toward the fresher sea. Gatsby’s eyes followed it momentarily (...) Slowly the white wings of the boat moved against the blue cool limit of the sky. (...)

“There’s sport for you,” said Tom, nodding. “I’d like to be out there with him for about an hour.”

Thank you.