The narrator, who is not named but simply called "Brother", recounts the life of his younger brother, William Armstrong, whom his brother named "Doodle". From then on, he was referred to as Doodle. Doodle is born a sickly child who is not expected to live (a small coffin is even made in the case of his death). His brother wanted someone who could run and jump and play with him, but resents having the weak and fragile Doodle instead.
Eventually, at the age of six, Doodle learns to walk with help from Brother. Encouraged by this, the brother decides to teach Doodle how to run, climb vines, swim, and even fight to prepare Doodle for school. However, almost a year after the plan was made, Doodle was far from accomplishing the goals by the nearing deadline.
One day, a big red bird appears in their garden, looking sick and tired. The boys' father identifies it as a scarlet ibis, a tropical bird that was blown off-course by a recent storm. When the bird dies, Doodle, pitying the creature, buries it. Afterwards, the boys go to nearby Horsehead Landing to continue Doodle's "training". On their way back to the house, Brother has Doodle practice rowing. A sudden rainstorm comes, and when they reach the riverbank, Doodle is tired and frightened. Brother, angry and frustrated that Doodle could not finish his training before school starts, runs ahead of Doodle, leaving the frightened boy behind. When Brother does not see Doodle, Brother returns for Doodle, only to find Doodle, lifeless, lying on the ground with blood flowing out of his mouth, staining his throat and shirt red. Doodle died like the scarlet ibis: red and far away from home.