On May 3, 1977, Lydia Lee, the middle child of the Lee family goes missing. After several days her body is dredged out of the town lake. Lydia's parents, James and Marilyn are horrified with their daughter's death. As the police investigate what happened, her parents both discover that contrary to their belief that Lydia was popular and doing well in school, she was actually a loner with almost no friends and that her grades had severely slipped.
The death of their child forces James and Marilyn to reflect back on their lives. James, the academically gifted child of Chinese immigrants spent his entire life yearning to belong and went on to become a university professor, where he met Marilyn in 1957 in a class he was teaching on American culture at Radcliffe. Meanwhile, Marilyn grew up disgusted by her homemaker mother and longed to become a doctor. When she met James and recognized the racist treatment he had been receiving, Marilyn immediately felt a kinship with him and the two began an affair. Discovering she was pregnant, she arranged for a quick marriage to James and was disgusted when her mother tried to stop the wedding after seeing that James was of Asian descent.
Marilyn intended to go back and continue her studies to become a doctor after her son, Nathan, was born, but after a second pregnancy with Lydia she remained a home-maker for eight years. After receiving news of her estranged mother's death, Marilyn returned to her childhood home to deal with her mother's things and had a crisis after realizing that she achieved the home-maker life her mother always desired for her. After briefly returning home, she abandons the family in order to pursue her academic studies without leaving a note. James believes that she has left because he and the children are Asian and she no longer wants to deal with the societal pressure of being outsiders.
Marilyn's absence lasts nine weeks, during which she realizes she is pregnant with a third child, Hannah. She returns home and realizes she will never have the will to abandon her children and pursue a career again. Instead she begins to encourage Lydia to become a doctor, aggressively training her in math and other sciences. During Marilyn's absence, James also began favouring Lydia and bullying Nath, whom James perceives is as friendless as James himself was in childhood. Nath becomes jealous of Lydia and one day pushes her into the lake even though she cannot swim. However, the ease with which she falls causes Nath to realize that Lydia is drowning under the weight of her parents' expectations. He rescues her and the two become close.
By the time they are teenagers, Lydia begins buckling under the weight of her mother's expectations and cannot keep up with the advanced courses her mother encourages her to take. She also is sick of faking she has friends for her father. She begins to hang out with Jack, her next door neighbor whom Nath hates, who has a reputation for deflowering town girls. Meanwhile, Nath, who has spent the past few years being ignored by his parents, is accepted into Harvard. Lydia is scared of being abandoned by Nath and hides his acceptance letters. When she is caught, a rift develops between Lydia and Nath. Lydia goes to Jack, hoping to sleep with him; however, Jack tells her he is in love with her brother and his reputation as a Lothario is a lie. Returning home, Lydia is determined to reveal her shortcomings to her father and mother. Tracing back her unhappiness to the time her brother pushed her into the lake, Lydia goes there late at night intending to jump into the water and swim back to shore.
In the present, Marilyn discovers that after Lydia's death, James has begun having an affair with one of his grad students, Louisa Chen, who is also of Chinese descent. She learns that he feels that Marilyn resents their marriage because he and the children are not white and he leaves her. Eventually, he returns and he and Marilyn slowly begin to reconnect. Nath, who still believes that Jack is responsible for Lydia's death, confronts him by the lake, punching him twice before Hannah, who has realized that Jack is in love with Nath, stops him. Nath falls into the lake, where he realizes that he will never be able to understand Lydia's death and achieves a modicum of closure.