The story starts with a prologue, telling the tale of young British aristocrat, Richard Devine, who is the son of a shipbuilding magnate, Sir Richard Devine. In an incidence of domestic violence, Richard's mother reveals to Sir Richard that his son was fathered by another man, Lord Bellasis. Sir Richard proceeds to threaten the mother's reputation if Richard does not leave and never come back. He leaves him to pack for a while, claiming that he will fetch his lawyer to alter his will so that Richard receives no inheritance. When Richard leaves, he comes across a murder scene: his biological father, Lord Bellasis has been murdered, and Richard witnesses Sir Richard walking away from the scene of the crime. The police come and lock up Richard, who now gives his name as Rufus Dawes (which is used for the remainder of the book), for the murder of Lord Bellasis. Additionally, Sir Richard returns home and dies straight away, possibly of a heart-attack, without altering his will. Rufus Dawes/Richard Devine never finds this out. Rufus is found not guilty of the murder but guilty of the robbery of the corpse and sentenced to transportation to the penal colony of Australia.
In 1827, Dawes is shipped to Van Diemen's Land on the "Malabar", which also carries Captain Vickers, who is to become the new commander of the penal settlement at Macquarie Harbour, his wife Julia and child Sylvia, Julia's maid, one Sarah Purfoy and Lieutenant Maurice Frere, Richard Devine's cousin, son of Sir Richard's sister, who would have inherited the fortune in Richard's place. It turns out that Sarah is on the vessel only to free her lover, John Rex. She organises a mutiny with the help of three other men: Gabbett, James "Jemmy" Vetch or "the Crow" and a man nicknamed "the moocher", while John Rex is in hospital with the fever. One night, a burning vessel is sighted and found out to be the Hydaspes; the ship on which Richard Devine is supposed to have sailed. The crew cannot be found. The following day, Dawes overhears the plans of the mutineers, but is taken to the hospital sick with the fever shortly afterwards. He manages to warn Captain Vickers and Doctor Pine about the plans. Sarah gets the Captain drunk and Frere otherwise busy but does not know about Vickers clandestinely doubling the guard. The mutiny is, therefore, unsuccessful, but Jemmy Vetch, who has understood that only Dawes could have betrayed them, gets his revenge by claiming that Dawes was the ring leader of the mutiny. Dawes is found guilty and receives a second life sentence.
In 1833, at Macquarie Harbour, Maurice Frere has come to deliver to Captain Vickers the news that the settlement at Macquarie Harbour is to be abandoned and the convicts to be moved to Port Arthur. He also attempts to befriend Sylvia, but the child resents him ever since witnessing his treatment of the convicts (especially that of Dawes, who went to the quarterdeck to return her ball). Rufus Dawes has been the victim of several assassination attempts at the convicts' hands, but has also attempted escape twice and has a long record of bad conduct and punishments. At the moment of Frere's arrival, he is in solitary confinement on Grummet rock, a small island before the coast. Dawes has managed to recognise Frere at the harbour and now, seeing the preparations for the abandonment of the settlement, mistakenly assumes that Frere has taken command. Rather than suffer Frere's treatment upon his return, he attempts to drown himself.
Meanwhile, it has been decided that Vickers should sail with the convicts and Frere follow with the brig "Osprey" with Mrs. Vickers and Sylvia, the pilot, five soldiers and ten convicts. Among the convicts is John Rex, who has again plans for mutiny. The convicts succeed in taking the boat, killing two soldiers, wounding one, and marooning him with the Vickers, the pilot and Frere. The pilot and the wounded man die shortly afterwards. One night, a man reaches their makeshift camp. It is Rufus Dawes, who has managed to swim to the settlement only to find it deserted. Although initially wary of him, the little community soon accepts Dawes, especially since he knows all kinds of ways to make their life more agreeable. Sylvia takes to him and Dawes soon does everything to please her, despite Frere's jealous attempts to win Sylvia's affection. It is also Dawes, who, after Sylvia mentioned the coracles of the Ancient Britons, plans and succeeds in building a boat out of saplings and goat hide. Although Frere promises Dawes a pardon, he nevertheless does not stop treating him like an inferior, at one point upsetting Dawes so much that he considers leaving on his own. Only Sylvia's writing in the sand, "Good Mr. Dawes", stops him. Through a hazard, Frere tells Dawes of the fate of his cousin and how narrowly he missed inheriting the Devine fortune. Dawes had not known about Sir Richard's death. Finally, they set to sea with Mrs. Vickers gravely ill and Sylvia soon also sick. After some time they are found by an American vessel at which point Frere takes the rudder of the boat and Sylvia in his arms.
By 1838, in Port Arthur, Mrs. Vickers has died. Sylvia has lost all her memories of the incident at Macquarie Harbour and knows only what she has been told about it. She is now a young woman of sixteen and engaged to Captain Maurice Frere, who has told the story of the mutiny in his own way: making himself the hero and claiming that Dawes attempted to murder all three of them. News arrives that the surviving mutineers of the Osprey have been captured and are to be tried at Port Arthur. Sarah Purfoy calls on Frere and begs him to speak in Rex's favour, saying that he left them food and tools. She threatens to expose Frere's previous affairs to Sylvia. Frere consents to her demands. Rufus Dawes is also brought down from Hobart to identify the captured men. At the trial, he sees Sylvia again and realises that she is alive: he had been informed of her death. He tries to speak his case but is not allowed to. The mutineers get away with life sentences.
Dawes escapes to see Sylvia again and begs her to speak, but in her amnesia she is afraid of him and calls for help. Dawes, too thunderstruck to leave, is immediately recaptured and sent back to Hobart. There, he meets the Reverend James North, a drunkard, whose failure to get up in time after a drinking night results in the death of a convict at the triangle, whom North had sworn to protect. Dawes is ordered to carry out the flogging and upon eventually refusing is flogged himself. Despite Dawes' initial hate for the man he considers to be a hypocrite, he is moved by North's begging for forgiveness and calling him "brother". The next time he asks to see the chaplain he finds that North, an enemy to the bishop for his impious vices, has been replaced by Meekin, a dainty man, who lectures him on his sins rather than attempting to console him.
John Rex seeks Dawes and tries to persuade him to join him in an escape, organised by Sarah Purfoy. Dawes refuses. Through luck, Rex starts talking about the Devines and about how he was once employed to find news of their son. Dawes, appalled, asks if he would still recognise the man and Rex understands all of Dawes' story. When shortly afterward a warder confuses them both, commenting on how much they look alike, Rex hatches another plan.
A few days later, Rex and another group of eight, led by Gabbett and Vetch, escape. It soon becomes apparent that Rex used the other men only as decoys. They get hopelessly lost in the bush and start eating one another, leaving only Gabbett and Vetch to struggle for not being the first to fall asleep. Later, Gabbett is found on a beach by the crew of a whaling vessel, with the half-eaten arm of one of his comrades hanging out of his swag. This part is based on a true story, that of Alexander Pearce.
Rex reaches Sydney and, soon becoming weary of Sarah, escaping her to go to London, where he presents himself as Richard Devine. Lady Ellinor accepts him as her son.
In Norfolk Island, by 1846, Reverend James North has been appointed prison chaplain. Shortly afterward, Captain Frere becomes Commandant of the Island, resolved to enforce discipline there. North, appalled at the horrible punishments inflicted but not really daring to interfere, renews his friendship with Dawes and also takes to Sylvia. Her marriage is an unhappy one. Frere has grown weary of his wife over the years and Sylvia married him only because she believed that she owed love to the man who allegedly saved her life. Dawes has also been on the Island for five years and again becomes Frere's target. Frere is resolved to break his opponent's spirit and finally succeeds after inflicting punishment upon punishment on him for several weeks. One night, Dawes and his two cell mates draw lots. The longest straw and old hand Blind Mooney is killed at the hands of the second, Bland, with Dawes as the witness. According to their plan, Bland and Dawes get sentenced to death.
North, in the meantime, has had to realise the true nature of his affection for Sylvia. At first, he attempts to keep away from her, but this unfriendliness is ill-received by Frere, who gets his revenge on the convicts open to North's words, especially on Dawes. One day, Sylvia has gone to see Dawes. She has not been able to stop thinking about him as she feels that there is more to the story than she knows. She finds Dawes on the "stretcher" and orders his release. Frere is furious when he learns about it and strikes Sylvia, despite North's presence. North has sent his resignation two months previously. Sylvia had already decided to go and see her father to escape the grievances of her life on the Island. The two admit their feelings for one another but decide to keep quiet until they can sail. North visits Dawes and learns the true story of the mutiny and rescue. He promises to tell Sylvia Dawes' story but does not.
Meanwhile, Sarah has found John Rex in London. He has led a life of debauchery, much to the disapproval of Lady Ellinor. Sarah threatens to denounce John if he does not introduce her as his wife. Ever since Rex wanted to sell the family house, Lady Ellinor's suspicions have reached the point where she attempts to test her alleged son. When she forbids him to sell the house, Rex says that it only is his right to do so. Lady Ellinor tells him that he has no rights to anything since Richard Devine was a bastard, the son of Lord Bellasis. Rex suddenly understands their strange resemblance: he is also a son of Lord Bellasis; his mother was a servant in his house. Rex confesses to the murder of Lord Bellasis, who laughed at him when told this story. Lady Ellinor promises Sarah to allow them to leave the country in exchange for information about her son. Sarah manages to get them both aboard a vessel bound for Sydney, but Rex dies of a stroke during the voyage.
Shortly before leaving, North visits Dawes to confess that he never talked to Sylvia because he is himself in love with her. Dawes tells the priest that he knows nothing about love and recounts his own story to illustrate his words. North confesses to having been the one who robbed the corpse of Lord Bellasis, as the Lord held proofs against North, who had been forging bank notes. Begging forgiveness again, North leaves in great confusion, forgetting his hat and cloak. Dawes manages to get out and on the boat in this disguise as the drunken warder has not closed his cell door. North observes him leaving and decides that it is for the best. The ship on which Dawes and Sylvia sail soon gets into a storm. Sylvia, seeking comfort from the reverend, finds Dawes in his place, and now remembers the past as the gale reaches its greatest force.
The next morning finds their entangled corpses on a piece of planking from the sunken ship.
List of locations described in the novel
- Macquarie Harbour Penal Station
- Hell's Gates
- Frenchmans Cap
- Port Arthur, Tasmania
- Island of the Dead 
- Eaglehawk Neck, Tasmania
- Norfolk Island