The story takes place in and around the apartment of Elizabeth and Gabriel and centers on the actions of their sons, Roy and John. Their family lives in the far northern part of Manhattan, only a short walk from the Washington Bridge, which crosses to the Bronx. On their block, emerging from the ground of a vacant lot between two houses, there is a large rock formation upon which neighborhood boys play and tend to get rowdy. The mystery of the rockpile fills Roy with curiosity, which is amplified by the supernatural stories the adults tell about it, somewhat in an effort to deter the children from playing on it. Among these stories, the narrator recounts one told by Roy and John's Aunt Florence, who lives in the Bronx, that the rockpile exists because "without it the subway cars underground would fly apart, killing all the people." Unlike Roy, Roy's brother John has little interest in the rockpile. For him, the possible consequences of playing there outweigh any pleasure he might find.
One afternoon, Elizabeth goes over to their neighbor Sister McCandless's apartment to chat. John and Roy sit on the fire escape where, if she wanted to, she could peek out the window and check on them. She's always warned them not to go messing around on the rockpile, but now that she's gone, Roy feels inspired to do just that. A few of his neighborhood friends pass by under the fire escape, and he tells his older brother John that he's going to go with them. John warns him not to. He warns him that his mother could come home at any time, and even more worryingly, his father is going to be home soon.
Roy goes anyway, leaving John alone on the fire escape to continue working on his homework. John doesn't tattle on Roy because he feels an allegiance to his brother, but he feels uneasy about letting him go. He knows he couldn't really stop him from going, but at the same time feels responsible. After Roy leaves, a gang fight breaks out on the rockpile, and boys start hitting each other and throwing rocks and cans and whatever else they can get their hands on. John sees Roy emerging from the pile with some friends, his shirt ripped but himself looking in fine spirits. Then, from somewhere on the rockpile, a tin can flies and hits Roy in the face, leaving a bloody gash under his eye. He starts screaming and crying and runs to his mother by Sister McCandless's building.
Elizabeth mildly panics, so Sister McCandless, older and calmer than Elizabeth, takes charge. Roy's wound is cleaned, disinfected, and bandaged. Once the wound is taken care of, Elizabeth and Sister McCandless begin asking questions about why Roy was outside in the first place. The questions are directed towards John, the older brother, who is supposedly responsible for his younger brother's actions. When the women ask John why he didn't tell them Roy went downstairs, he insisted, "He said he'd be back in five minutes," but to McCandless, this is an insufficient response. She says, "You’s the man of the house, you supposed to look after your baby brothers and sisters—you ain’t supposed to let them run off and get half-killed." She predicts a reckoning when Gabriel, their father, returns home.
Immediately as Gabriel returns home, the narrator implies Gabriel's favoritism towards Roy through exposition with the fact that John is Elizabeth's only child from another man: "Then she heard the front door open and close—too loud, Delilah raised her voice, with an exasperated sigh Elizabeth picked the child up. Her child and Gabriel’s, her children and Gabriel’s: Roy, Delilah, Paul. Only John was nameless and a stranger, living, unalterable testimony to his mother’s days in sin." This suggestion is immediately reinforced by the text; Gabriel is exceedingly gentle with Roy and tolerant of his crying and screaming at a wound that is, in all likelihood, very minor.
When Gabriel starts asking John questions about what happened, Elizabeth steps in and shoulders the responsibility for letting Roy out of her sight in an obvious attempt to protect John from Gabriel's ire. Eventually, despite Elizabeth's attempts to divert the blame, Gabriel asks where John was while his brother was playing outside. When John doesn't answer, Gabriel threatens to beat him. Elizabeth stands firm in her position that Gabriel will not beat John over this, that he couldn't do anything to stop Roy from disobeying her and playing on the rockpile because Roy takes after Gabriel in his stubbornness. She says, “You ain’t going to take no strap to this boy, not today you ain’t. Ain’t a soul to blame for Roy’s lying up there now but you—you because you done spoiled him so that he thinks he can do just anything and get away with it. I’m here to tell you that ain’t no way to raise no child. You don’t pray to the Lord to help you do better than you been doing, you going to live to shed bitter tears that the Lord didn’t take his soul today.".
After she makes her proclamation, she trembles in place at the possible consequences. Gabriel's face is full of rage and hatred for her, hatred "so deep as to become insupportable in its lack of personality." She then takes the infant, Delilah, out of John's arms, and Gabriel appears to soften, if only slightly, at the notion that Elizabeth is his "helpmeet given by the Lord." The story concludes with the image of John, saved for now from a beating, scrambling to pick up his father's lunchbox, "bending his dark head near the toe of his father’s heavy shoe."
"The Rockpile" is a story of childhood, the conventions of patriarchal households, and blood allegiances, and contains biblical allusions and Christian symbolism. The titular formation of rocks that occurs in a vacant lot between two houses captures the imagination of Roy and looms large in his mind as a manifestation of supernatural and even religious forces. The explanation his Aunt Florence offers for the rockpile's existence is that "without it the subway cars underground would fly apart, killing all the people," thus attributing to the rockpile a godlike role in protecting and maintaining the lives of "all the people." His perception of the rockpile is reinforced when one of the neighborhood boys, Richard, drowns in the river. Richard's father carries his lifeless body down their street to their home, and Baldwin describes Roy and John watching "the small procession [disappear] within the house which stood beside the rockpile" (Baldwin). Thus, the rockpile is associated with absorbing the dead, much like an afterlife or deity. In this way, the rockpile figures religiously into Roy's worldview as something that both preserves life and brings forth the dead.
Roy seems to conflate the towering presence of the rockpile with his father's own towering presence in his life as a figure of ultimate authority: benevolent and gentle towards him, but capable of doing great harm. John's aversion to playing on the rockpile likely stems from the fact that he knows Gabriel seeks excuses to scold and beat him because he's not his biological son; conversely, Roy knows his father favors him, and this favor allows him to entertain his fascination with the rockpile and dare to play on it in defiance of his mother's instructions. His father's touch reminds Roy of "the height, the sharp, sliding rock beneath his feet, the sun, the explosion of the sun, his plunge into darkness and his salty blood" perhaps because it is this gentle touch from his father that allowed him, in the first place, to confidently play on the rock.
Elizabeth feels a particular need to defend John from Gabriel because he is her blood relation alone. She feels singularly responsible for his wellbeing because to Gabriel, "John was nameless and a stranger, living, unalterable testimony to his mother’s days in sin." His status as "nameless" refers to his surname; Gabriel may object to John taking his last name because he is not his biological son. Elizabeth feels that John is a product of her "days in sin," making her that much more responsible for his wellbeing if she associates his existence and the precarious role he occupies in Gabriel's household as a result of her own actions.
Every name in the nuclear family of "The Rockpile" has a strong biblical association (except Roy, which has various origins, some meaning "King"). Gabriel is an archangel who, in the New Testament of the Bible, visits Elisabeth, who up to that point in her life was barren. God, through Gabriel, gave Elisabeth the ability to conceive John the Baptist. Delilah appears in the Old Testament, in a chapter of Judges, and the Apostle Paul spread the word of Christ through his Gospel. The most significant triangle of names in the story is that of John, Elizabeth, and Gabriel. One could argue that Gabriel, by taking John into his home, gives him life. But Gabriel's support comes at a price for John, who lives in fear that his actions, or the actions of his brother, might cause Gabriel to unfairly punish him because of some perceived debt he owes by having been fathered by another man.
The story also alludes to a rural past for Elizabeth and Sister McCandless. When Roy runs crying and injured into his mother's arms, Baldwin writes that "Sister McCandless, bigger, calmer, took him from the man and threw him over her shoulder as she once might have handled a sack of cotton." Later, when Gabriel demands from John an answer as to his role in keeping an eye on Roy, Baldwin notes Elizabeth's observation of John: "The child stared at the man in fascination and terror—when a girl down home she had seen rabbits stand so paralyzed before the barking dog." Both of these accounts allude to a rural, Southern past that, in the time of this story, would suggest for its characters a likely past of sharecropping, indentured servitude, or a recent memory of slavery, at most one generation removed from its cruelty.