After a month of driving, the family finally makes it to Welch, West Virginia. After meeting Erma, her father’s mother who refuses to be called ‘Grandma,' Grandpa Walls, and Uncle Stanley, Jeannette is shocked to find no resemblance between them and her father. Welch is a picture of destitution and unemployment, though in earlier days it thrived as a coal mining town. Rex takes his family on a tour of the town and explains that the very first food stamps were handed out in Welch after hard times came to its people in the 1950’s. Rose Mary sees the destitution of the town as an opportunity to establish herself as an artist since there would be no competition nearby.
Brian and Jeannette are placed in special classes for children with learning disabilities at Welch Elementary School because the principal is unable to understand their accents and they are unable to understand his. The students and teachers at Welch blame Jeannette for thinking that she is better than them. At recess, Jeannette is jumped by a crowd of girls, led by a bully named Dinitia Hewitt, but she does not tell her parents about it.
One day, Jeannette helps a young African American boy escape a dog attack. She carries him home and sees Dinitia watching her from across the street. After that, Dinitia begins to befriend Jeannette. Erma and Stanley, having prejudice towards black people, question Jeannette's choice of friends and warn her that others will consider her a "nigger lover." Jeannette and her siblings are forced to stay with their grandparents alone when Rex and Rose Mary decide to return to Phoenix to retrieve some of their things and to check on the house. While they are gone Erma tries to force herself onto Brian, claiming that his pants need mending. A fight ensues when Lori tries to get Erma to stop. When Rex returns and hears about the incident he is angry at the children and doesn't appear to care about what happened to Brian. His reaction causes Jeannette to wonder whether he was abused similarly as a child. After this, the family is kicked out of Erma and Ted's house and they must move into a dilapidated apartment at 93 Little Hobart Street which, in addition to needing numerous structural repairs, has no indoor plumbing.
Jeannette tries to fix up the home but her family is not invested in the efforts. They all seem to think that the house is damaged beyond repair. Brian and Jeannette also resume work on the Glass Castle, with the encouragement of their father. But the hole they dig for the foundation of the Glass Castle instead becomes a place for the Walls' to dump their trash, since they are unable to pay for garbage to be picked up from their home. Jeannette befriends a girl named Kathy Pastor whose mother, Ginnie Sue, works as a prostitute. When she visits, Jeannette intends to ask Ginnie Sue about her life as prostitute but she forgets and instead tells them about her life in California.
Jeannette and Brian are teased by another kid in the neighborhood, Ernie Goad. Ernie calls their family garbage and assaults Brian and Jeannette with rocks. The siblings retaliate and chase Ernie and his gang away for good by dumping a pile of rocks on them from a building roof. Rex continues to drink heavily. One night he returns with a gash on his forearm which Jeannette tries to sew up. He begins disappearing from the house for days without returning and Jeannette finds it hard to continue having faith in him.
That winter is particularly challenging because the family cannot afford to heat the house. In an attempt to start a fire with kerosene, Lori accidentally gets burned. Brian and Jeannette treat her wounds with snow but Lori develops painful blisters down her thighs. Erma Wells dies that winter. Rose Mary believes that her alcoholism was responsible for her death and consequently defines her death as suicide. Rex is upset by his family's lack of remorse for his mother's death and he disappears for days. Jeannette is sent to fetch him from one of the local pubs. Shortly after Erma's death, Uncle Stanley burns the house down after he falls asleep smoking a cigarette. He and Ted survive and are forced to relocate to a two-room apartment. On occasion, the Walls visit their apartment to use the running water. Jeannette stops making this visit however after Uncle Stanley touches her inappropriately while holding his privates. Rose Mary tells Jeannette to calm down and says sexual assault is a "crime of perception."
Jeannette and Brian discover a two-carat diamond ring in the yard outside the house. They show their mother and urge her to sell it so they can use the money for food, housing payments, and other essentials, but Rose Mary insists on keeping the ring to replace her wedding ring and to boost her self-esteem. Jeannette, who has been studying the livelihoods of other families in their neighborhood, tells Rose Mary that she needs to leave Rex so that the family can qualify for welfare. Rose Mary refuses and chastises Jeannette for losing hope in her father. Jeannette says that if she will not leave Rex then she should at least get another teaching job.
Rose Mary does not get a job right away. She only acquiesces after a child-welfare officer visits the house to see if dependent children are being neglected. Jeannette is the only one home when he arrives and she dodges his questions and tells him to return when her parents are home. She is frustrated and afraid by the visit and fears that the child-welfare officer will split up her siblings.
Although Rose Mary begins teaching regularly at Davy Elementary School just outside of Welch, the family continues to have money troubles. The money runs out every month and the children are once again forced to root through the garbage for food or subsist on nothing at all. Jeannette begins seventh grade at Welch High School that fall. She spends some time with Dinitia but not too much, because of persisting negative attitudes about racial mixing. Jeannette notices that Dinitia has changed. Later, Dinitia finds out she is pregnant and stops going to school. Jeannette discovers that Dinitia was sent to jail for stabbing her mother's boyfriend, who is likely the father of Dinitia's child and the reason for her change in demeanor.
That school year Jeannette begins working for the school newspaper, The Maroon Wave, where she first discovers her passion for journalism. Her faculty adviser, Miss Jeannette Bivens, was Rex's old English teacher who convinced him to stay in school and get his diploma when he was a high school student. At the newspaper Jeannette also has access to news stories from around the country and she gains a better understanding of the world.
Feeling badly about Maureen's poor upbringing, Jeannette convinces Brian and Lori to save up money to buy Maureen a gift for her seventh birthday. Maureen has grown up to a be a beautiful, blond-haired, blue-eyed girl and she spends a lot of time at friends' houses. Many of her friends have Pentecostal parents who look down on the Walls' parenting habits and Maureen adopts their beliefs, calling herself "born-again" on multiple occasions. That summer, Rose Mary goes away to renew her teaching certification and Lori is admitted to a government-run summer program for enriched children. Jeannette is left in charge of the house and she is determined to stick to a budget and feed Maureen and Brian. However, Jeannette is unable to resist her father's pleas for money and she loans him $30 dollars within a week. To earn back the money, Rex takes Jeannette out to a roadside bar where he swindles a man named Robbie in pool using Jeannette as a distraction. Afterward, Jeannette feels used by her father but when she expresses this Rex says that she was in no real harm, it was just like when he threw her into the Hot Pot; she thought she would sink but he knew she could swim.
When Lori and Rose Mary return from their summer locations both appear changed. Lori is determined to get away from Welch and her family after discovering what it is like to be treated "normal" by the kids at camp. Rose Mary has decided that she won't be teaching again but will resume her life as an artist. When Jeannette talks back to her mother, telling her that she cannot quit her teaching job because they need the money, Rose Mary asks Rex to discipline her. Jeannette tells him that he too has been shirking his parental duties and Rex gives her a whipping with his belt. After this, Jeannette resolves never to receive a whipping again and she begins to save money for her escape out of Welch.
After two filmmakers arrive in Welch as part of a government cultural enrichment program, Lori is inspired to move to New York. The filmmakers, Ken Fink and Bob Gross, insist that she will have better chances of becoming a successful artist if she moves to Manhattan. When Lori shares this plan with Jeannette, Jeannette lets her know about the escape fund and both of the girls begin contributing to it. They keep their money in a piggy bank which they name Oz. That winter, Rex arrives at home with a gold Cadillac Coupe DeVille, which he names Elvis. He won the car in a game of poker. Although Jeannette thinks they should sell the car to pay for essentials, she enjoys the feeling of riding in it and is reminded of how easy it is to move once you set your mind on doing so.
Lori begins work on her art school portfolios but she is unsuccessful. She spills coffee on he first set of work and Rex, drunk, deforms a clay bust Lori had made of William Shakespeare. Lori becomes even more defeated when Jeannette discovers that Rex has stolen all of the escape money from Oz. After this Lori stops talking to Rex. They begin saving again but they are not able to earn nearly as much money and Lori loses morale. That summer, a family Jeannette babysits for offers her a job in Iowa, where they plan to move. Jeannette asks if Lori can take the babysitting job instead and receive a bus ticket to New York after the summer ends. The family agrees and Lori sets off for Iowa, without saying goodbye to her father.
Jeannette becomes increasingly involved in the school newspaper and is promoted to editor-in-chief by her junior year of high school. When Chuck Yeager visits Welch High School she is even permitted to conduct a one-on-one interview with him. Students stop making fun of Jeannette and begin to treat her like the cheerleaders and varsity athletes. Jeannette resolves to leave Welch after classes end to join Lori in New York City. Her father tries to entice her to stay, promising to resume work on the Glass Castle but Jeannette tells him not to bother building it for her sake; she is leaving Welch even if she has to walk to New York City. Rex walks his daughter to the bus station to see her off and he gives her his favorite jackknife for protection. From the bus, Jeannette turns around to wave goodbye to her father and wonders if she will indeed make it in the big city.
Racial difference becomes more pronounced when the Walls move to Welch. There, the expectation is that the races will remain separate. Though segregation is no longer legal, public facilities, like the swimming pool, maintain defacto separation. The African Americans arrive during the morning when there is no entrance fee while white people swim during the afternoon when the fee is charged. Jeannette develops a friendship with Dinitia, a black girl, which causes trouble with her Grandmother Erma and her Uncle Stanley. Jeannette defends her position that the races are not different from each other which greatly upsets her grandmother.
The move from the west to the east coast is accompanied by more than a change in racial dynamics. The Walls have to adjust to a number of things characteristic of their new environment, including the harsh winter months. The loss of the nature and warmth of the west coast, impacts the general happiness of the Walls family. Rex is defeated by having to return to the place he spent his entire childhood trying to escape, and the Walls children are ready to leave almost as soon as they arrive. Welch symbolizes the end of adventure for the Walls family, and the place from which they must all eventually escape.
93 Little Hobart Street is most symbolic of the depression that accompanies the family's move to Welch. From the outset, the house is shoddy and in desperate need of repair. However, Jeannette is the only one who tries to fix it; all other members have given up fixing or improving the house. Everyone appears hopeless that their conditions will improve and powerless to change them.
Finally, Jeannette begins to recognize that her father continually the family down. Once the only child who believed in him, she loses hope in Rex as his drunkenness worsens. When Jeannette asks Rose Mary to leave Rex so that the family can qualify for welfare, Rose Mary is shocked and she admonishes Jeannette for losing hope in her father.
The dream of the Glass Castle is also destroyed during this time. Brian and Lori begin digging a foundation for the home Rex had designed for his family, but the foundation instead becomes a ditch for garbage that the family cannot afford to have picked up. The idea and dream of the Glass Castle as a physical structure is also destroyed when this happens. Jeannette realizes that it will never be built. Yet, the Glass Castle as a symbol and an abstract hope lives on.