Sherlock Holmes, suffering from boredom due to a want of cases, returns home from a walk with Dr. Watson in the early spring of 1888 to find he has missed a visitor, but that the caller has left his pipe behind. From it, Holmes deduces that he was disturbed of mind (because he forgot the pipe); that he valued it highly (because he had repaired, rather than replaced, it when it was broken); that he was muscular, left-handed, had excellent teeth, was careless in his habits and was well-off.
None of these deductions is particularly germane to the story: they are merely Holmesian logical exercises. When the visitor, Mr. Grant Munro (whose name Holmes observed from his hatband) returns, Holmes and Watson hear the story of Munro's deception by his wife Effie. She had been previously married in America, but her husband and child had died of yellow fever, whereupon she returned to England and met and married Munro. Their marriage had been blissful — "We have not had a difference, not one, in thought, or word, or deed," says Grant Munro — until she asked for a hundred pounds and begged him not to ask why. Two months later, Effie Munro was caught conducting secret liaisons with the occupants of a cottage near the Munro house in Norbury.
Grant Munro has seen a mysterious yellow-faced person in this cottage. Overcome with jealousy, he breaks in and finds the place empty. However, the room where he saw the mysterious figure is very comfortable and well-furnished, with a portrait of his wife on the mantelpiece.
Holmes, after sending Munro home with instructions to wire for him if the cottage was reoccupied, confides in Watson his belief that the mysterious figure is Effie Munro's first husband. He postulates that the husband, having been left in America, has come to England to blackmail her.
After Munro summons Holmes and Watson, the three enter the cottage, brushing aside the entreaties of Effie Munro. They find the strange yellow-faced character; Holmes peels that face away, showing it to be a mask and revealing a young black girl. Effie Munro's first husband was John Hebron, a black man; he did die in America, but the couple's daughter Lucy survived. Afraid that Grant Munro would repudiate his love for her if he knew she was mother to a mixed race child, she had endeavored to keep Lucy's existence hidden. Overcome with desire to see her child again, Effie Munro used the hundred pounds to bring Lucy and her nurse to England and installed them in the cottage near the Munro house.
Both Watson and Holmes are touched by Munro's response. Watson observes:
...when [Munro's] answer came it was one of which I love to think. He lifted the little child, kissed her, and then, still carrying her, he held his other hand out to his wife and turned towards the door. "We can talk about it more comfortably at home," said he. "I am not a very good man, Effie, but I think I am a better one than you have given me credit for being."
"Watson, if it should ever strike you that I am getting a little overconfident in my powers, or giving less pains to a case than it deserves, kindly whisper 'Norbury' in my ear, and I shall be infinitely obliged to you."