The story begins with the reminiscences of Leo Colston, an elderly man looking back on his childhood with nostalgia. Leo, in his mid-sixties, is looking through his old things. He chances upon a battered old red collar box. In it he finds a diary from 1900, the year of his thirteenth birthday. He slowly pieces together his memory as he looks through the diary.
Impressed by the astrological emblems at the front of the book, young Leo combines them in his mind with the idea that he is living at the turn of the 20th century. The importance of his boarding school's social rules is another theme. Some of the rougher boys steal his diary, reading and defacing it. The two oldest bullies, Jenkins and Strode, beat him at every opportunity. He devises some "curses" for them in the pages of the book, using occult symbols and Greek letters, and placing the book where they will find it. Subsequently both boys venture onto the roof of one of the school buildings, fall off and are severely injured. This leaves him greatly admired by the other boys, who think that he is a magician – something that he comes to half-believe himself.
The greater portion of the text concerns itself with Leo's past, particularly the summer of 1900, spent in Norfolk, England, as a guest at Brandham Hall, the luxurious country home of his schoolfriend Marcus Maudsley. Here the young Leo, on holiday from boarding school, is a poor boy among the wealthy upper class. Leo's comparatively humble background is obvious to all and he does not really fit in there; however, his hosts do their best to make him feel welcome, treating him with kindness and indulgence. When Marcus falls ill, Leo is left largely to his own devices. He becomes a secret "go-between" for Marian Maudsley, the daughter of the host family, and nearby tenant farmer Ted Burgess. At first, Leo is happy to help Marian because she is kind to him and he has a crush on her. Besides, Leo is initially ignorant of the significance or content of the messages that he is asked to carry between Ted and Marian. Leo is a well-meaning and innocent boy, so it is easy for the lovers to manipulate him.
The fact that Ted comes from a much lower social class than Marian means there can be no possible future in the relationship because of the social taboos involved. Although Marian and Ted are fully aware of this, Leo is too naïve to understand why the lovers can never marry. The situation is further complicated by the fact that Marian is about to become engaged to Hugh, Viscount Trimingham, the descendant of the area's nobility who formerly resided in Brandham Hall. Together, these factors make Marian's secret relationship with Ted highly dangerous for all parties concerned.
Later, Leo acts as an interceptor, and occasional editor, of the messages. Eventually, he begins to comprehend the sexual nature of the relationship between Marian and Ted, and feels increasingly uncomfortable about the general atmosphere of deception and risk. Leo tries to end his role as go-between, but comes under great psychological pressure and is forced to continue. Ultimately, Leo's involvement as messenger between the lovers has disastrous consequences. The trauma which results when Marian's family discover what is going on leads directly to Ted's shotgun suicide.
In the epilogue the older Leo tells the reader the consequences of this summer. The experience profoundly affects Leo, leaving him with permanent psychological scars. Forbidden to speak about the scandal, he feels he must not think of it either; and since nearly everything reminds him of it, he shuts down his emotions, leaving room only for facts. He subsequently grows up to be an emotionally detached adult who is never able to establish intimate relationships. He succeeds in repressing the memories until the diary unlocks them. Now looking back on the events through the eyes of a mature adult, he is fully aware of how the incident has left its mark on him. In a final twist to the story, 52 years later, Leo returns to Brandham. There he meets Marian's grandson and finds Marian herself living in a cottage – the place she had always told people she was going when she was really having clandestine meetings with Ted. Brandham Hall has been let out to a girls' school. Lord Trimingham married Marian, but died in 1910, and Marcus and his brother Denys were killed in the First World War. In the end, an elderly Marian Maudsley persuades Leo to act as a go-between for her once more.