Narrating in the form of a bildungsroman, an elderly man, Stephen Wheatley, reminisces about his life during the Second World War as he wanders down the now modernised London cul-de-sac that he once called home.
Now a young boy, Stephen, regularly bullied at school and bored with his home life, is informed by his best friend Keith Hayward, a snobbish and domineering neighbour, that Keith's mother is an undercover operative working for the Germans. As the two boys spy on Mrs. Hayward from a hiding place in the hedges, they notice her unusual daily routine: leaving Keith's house with a picnic basket full of food, tapping on Auntie Dee's (Mrs. Hayward's sister, next-door-neighbour and best friend, whose husband, Uncle Peter, is away in the RAF) window, and walking through to the end of the cul-de-sac where she disappears into the nearby town. When the boys follow her, they cannot find her in any of the shops; and when they get back to their hiding place, Mrs. Hayward is already ahead of them, walking back into Keith's house.
When snooping in Keith's mother's room, they find her diary which contains a small 'x' marked on a day of every month (however this is in reference to her menstrual cycle). The boys' naïveté leads them to believe that 'x' is another secret agent that Mrs. Hayward has meetings with each month. One day, the boys realise that Keith's mother does not turn right into the town every day, but instead turns left into a grimy tunnel that leads to a disused field. Later that night, Stephen goes through the tunnel and finds a box in the field that contains pack of cigarettes. When Keith opens the packet, a slip of paper pops out with a single letter written on it:X. One night Stephen sneaks out to the tunnel and goes to the box once again. Inside this time were some clean clothes. As he is looking through them, somebody appears behind him. Stephen is too scared to turn around and holds his breath hoping that he isn't noticed. Still holding a sock, Stephen runs away as soon as he cannot hear the sound of breathing behind him. His family are outside looking for him and are furious.
The next day, when Keith is doing homework, Mrs. Hayward visits Stephen in his hiding place in the bushes and tells him that she knows he is following her, and that he should stop now before he gets hurt. Despite her warnings, Stephen, not telling him about Mrs. Hayward's warnings, shows Keith the sock and tells him that they need to uncover the truth before Keith's mother's next meeting with 'x'.
The next day, the boys revisit the field where they find the box empty. A few feet ahead of them they see something hiding under an iron sheet – a vagabond. Keith and Stephen take bars and smash at the sheet until finally realising they may have killed the vagabond. They run and bump into Keith's mother in the tunnel. She holds back Stephen and tells him since he is not going to stop spying on her, he will have to do her favours for the man in the field. Stephen realises that Mrs. Hayward is not a German spy, but in fact helping the vagabond whom she has taken under her bosom.
While taking eggs and milk to the tramp, Stephen discovers he is dying, and is asked to give a silk map to Mrs. Hayward to show his love for her. Stephen, however, is too scared to do so, and later that night sees police taking the vagabond away on a gurney; his face badly mutilated after being hit by a train. Fifty years later, Stephen ties up the loose ends: explaining that the vagabond was in fact Uncle Peter who had gone AWOL and was carrying out an affair with Keith's mother while dying from war wounds. As well as this, it turns out that there was a German spy living in the cul-de-sac: Stephen's father, although he was actually working for the English.
A subplot is also included in the novel, where Stephen finds comfort in Barbara Berrill – a girl Stephen's age living in his neighbourhood – who is used as a plot device for revealing very important information that helps Stephen understand the mysteries he is uncovering.