Sherlock Holmes, suffering from boredom due to a want of cases, returns home from a walk with Dr. Watson early in spring to find he has missed a visitor but that the caller has left his pipe behind. From it, Holmes deduces that he was disturbed of mind (because he forgot the pipe); that he valued it highly (because he had repaired, rather than replaced it, when it was broken); that he was muscular, left-handed, had excellent teeth, was careless in his habits and was well-off.
None of these deductions is particularly germane to the story: they are merely Holmesian logical exercises. When the visitor, Mr. Grant Munro (whose name Holmes observed from his hatband) returns, Holmes and Watson hear the story of Munro's deception by his wife Effie. She had been previously married in America, but her husband and child had died of yellow fever, whereupon she returned to England and met and married Munro. Their marriage had been blissful—"We have not had a difference, not one, in thought, or word, or deed," says Grant Munro—until she asked for a hundred pounds and begged him not to ask why. Two months later, Effie Munro was caught conducting secret liaisons with the occupants of a cottage near the Munro house in Norbury. Grant Munro has seen a mysterious yellow-faced person in this cottage. Overcome with jealousy, he breaks in and finds the place empty. However, the room where he saw the mysterious figure is very comfortable and well furnished, with a portrait of his wife on the mantelpiece.
Holmes, after sending Munro home with instructions to wire for him if the cottage was reoccupied, confides in Watson his belief that the mysterious figure is Effie Munro's first husband. He postulates that the husband, having been left in America, has come to England to blackmail her.
After Munro summons Holmes and Watson, the three enter the cottage, brushing aside the entreaties of Effie Munro. They find the strange yellow-faced character, and Holmes peels the face away, showing it to be a mask, and revealing a young girl who is half-black. It is then revealed that Effie Munro's first husband was John Hebron, an African-American lawyer; he did die in America, but their daughter, Lucy, survived. At the time Effie came to England, she could not bring Lucy with her, as the child was still ill enough that the transition might have exacerbated her state. But later, after she and Grant were married, Effie got word that Lucy was alive and well. Upon hearing of this, Effie became overcome with desire to see her child again, so she asked for the hundred pounds and used it to bring Lucy and her nurse to England, and installed them in the cottage near the Munro house. She feared, however, that Grant might stop loving her if he found out that she was the mother of a mixed race child, so all the while, she had made every endeavour to keep Lucy's existence a secret.
Both Watson and Holmes are touched by Munro's response. Watson observes:
...when [Munro's] answer came it was one of which I love to think. He lifted the little child, kissed her, and then, still carrying her, he held his other hand out to his wife and turned towards the door. "We can talk about it more comfortably at home," said he. "I am not a very good man, Effie, but I think I am a better one than you have given me credit for being."
Holmes excuses himself and Watson, and, that evening, after they have returned to Baker Street, says:
"Watson, if it should ever strike you that I am getting a little overconfident in my powers, or giving less pains to a case than it deserves, kindly whisper 'Norbury' in my ear, and I shall be infinitely obliged to you."