The story begins with young Kate sewing a quilt with her aunt Mrs May. As they knit the quilt, Mrs May tells Kate about her brother who told her about the Borrowers. She begins by telling her the story of fourteen-year-old Arrietty Clock who lived under the floorboards of a house with her father Pod and mother Homily. As Borrowers, they survive through Pod's "borrowing" of items from the "human beans" who live in the home above the floor. But one day Pod came home shaken after borrowing a toy tea cup. After sending Arrietty to bed, Homily finds out that he has been "seen" by a human bean: a boy who had been sent from India to live with his great-aunt while recovering from an illness. Remembering the fate of their niece Eggletina, who wandered away and never returned after her father had been seen and the big people had brought in a cat, Pod and Homily decide to tell Arrietty. In the course of the ensuing conversation, Homily realizes that Arrietty ought to be allowed to go borrowing with Pod.
Several days later, Pod and Arrietty go on a borrowing trip to retrieve fibres from a doormat for a scrubbing brush. Arrietty wanders outside where she meets the Boy, and develops a friendship with him. At one point, Arrietty tells the Boy that there cannot be very many of his kind but there are many of her kind. He disagrees and tells her of times when he had seen hundreds and even thousands of big people all in one place. Arrietty realizes that she cannot prove that there are any other Borrowers left in the world besides herself and her parents, and is upset. The Boy offers to take a letter to a badger sett two fields away where her Uncle Hendreary, Aunt Lupy, and their children are supposed to have emigrated. On a later borrowing trip, she manages to slip the letter under the doormat where the Boy agreed to look for it.
Meanwhile, Arrietty has learned from Pod and Homily that they get a "feeling" when big people approach. She is concerned that she didn't have a feeling when the Boy approached, so she practises by going to a certain passage over which the cook, Mrs Driver, often stands. She overhears Driver and the gardener, Crampfurl, discussing the Boy. Mrs Driver is annoyed that the boy continually disturbs the doormat and Crampfurl is concerned about him after seeing him in a field calling for "Uncle something", after the boy had asked him if there were any badger setts in the field. Crampfurl is convinced the boy is keeping a pet ferret.
Arrietty becomes anxious and sets off on her own to find the Boy. As it turns out, he did find her letter, delivered it, and returned with a response: a mysterious note asking her to tell Aunt Lupy to come back. Pod then discovers Arrietty talking to the Boy and takes her home. Pod and Homily are frightened because the Boy will probably figure out where they live. They turn out to be right, but the Boy, instead of wanting to harm them, brings them gifts of some doll's house furniture from the nursery. They experience a period of "borrowing beyond all dreams of borrowing" as the Boy offers them gift after gift. In return, Arrietty is allowed to go outside and read aloud to him.
Mrs Driver, in the meantime, notices a few items missing and thinks someone is playing a joke on her. She stays up late and catches the boy bringing his nightly gift to his new friends. She sees the Borrowers and finds their home. The Boy tries to rescue the Borrowers, but Mrs Driver locks him in the nursery. At the end of three days, the Boy is to be sent back to India. Mrs Driver cruelly takes him to the kitchen before he goes to see the ratcatcher to smoke the Borrowers out of their home. The Boy manages to slip away and break off the grating outside. However, he never gets to see the Borrowers' escape since the cab comes to take him away before he has a chance.
His sister (a young Mrs May, the narrator at the beginning of the book) later visits the home herself, goes to the badgers' sett, and leaves gifts there, which are gone the next time she checks. However, the novel ends when she tells Kate that when she returns to the badgers' sett she finds a book she believes to be Arrietty's book of "Memoranda" – and that the handwritten es look like a crescent moon with a stroke in the middle, the same way her brother used to write them.