Who accused the friends on trespassing ? What was the outcome ?

From the text:

We had just commenced the third course—the bread and jam—when a gentleman in shirt-sleeves and a short pipe came along, and wanted to know if we knew that we were trespassing. We said we hadn’t given the matter sufficient consideration as yet to enable us to arrive at a definite conclusion on that point, but that, if he assured us on his word as a gentleman that we were trespassing, we would, without further hesitation, believe it.

Finally Harris challengs the man to through them over. The man goes to consult his master but never returns.