Pound was born in a small, two-storey house in Hailey, Idaho Territory, the only child of Homer Loomis Pound (1858–1942) and Isabel Weston (1860–1948). His father had worked in Hailey since 1883 as registrar of the General Land Office.
Both parents' ancestors had emigrated from England to North America in the 17th century. On his mother's side, Pound was descended from William Wadsworth (1594–1675), a Puritan who emigrated from England to Boston on the Lion in 1632. The Wadsworths married into the Westons of New York. Harding Weston and Mary Parker were the parents of Isabel Weston, Ezra's mother. Harding apparently spent most of his life without work, so his brother, Ezra Weston and his wife, Frances, looked after Mary and Isabel's needs.
On his father's side, the immigrant ancestor was John Pound, a Quaker who arrived from England around 1650. Ezra's grandfather, Thaddeus Coleman Pound (1832–1914), was a retired Republican Congressman for northwest Wisconsin who had made and lost a fortune in the lumber business. Thaddeus's son Homer, Pound's father, worked for Thaddeus in the lumber business, until Thaddeus secured him the appointment as registrar of the Hailey land office. Homer and Isabel married the following year, and Homer built for her a home in Hailey. Isabel was unhappy in Hailey and took Ezra with her to New York in 1887 when he was 18 months old. Homer followed and in 1889 found a job as an assayer at the Philadelphia Mint. The family moved to Jenkintown, Pennsylvania and in 1893 bought a six-bedroom house in Wyncote.
Pound's education began in a series of dame schools, some of them run by Quakers: Miss Elliott's school in Jenkintown in 1892, the Heathcock family's Chelten Hills School in Wyncote in 1893, and the Florence Ridpath school from 1894, also in Wyncote. His first publication ("by E. L. Pound, Wyncote, aged 11 years") was a limerick in the Jenkintown Times-Chronicle about William Jennings Bryan, who had just lost the 1896 presidential election: "There was a young man from the West, / He did what he could for what he thought best; / But election came round, / He found himself drowned, / And the papers will tell you the rest."
Between 1897 and 1900 Pound attended Cheltenham Military Academy, sometimes as a boarder, where he specialized in Latin. The boys wore Civil War-style uniforms and besides Latin were taught English, history, arithmetic, marksmanship, military drilling and the importance of submitting to authority. Pound made his first trip overseas in the summer of 1898 when he was 13, a three-month tour of Europe with his mother and Frances Weston (Aunt Frank), who took him to England, Belgium, Germany, Switzerland and Italy. After the academy he may have attended Cheltenham Township High School for one year. In 1901 at the age of 15, he was admitted to the University of Pennsylvania's College of Liberal Arts.
He would write in 1913, in "How I Began":
I resolved that at thirty I would know more about poetry than any man living ... that I would know what was accounted poetry everywhere, what part of poetry was 'indestructible', what part could not be lost by translation and – scarcely less important – what effects were obtainable in one language only and were utterly incapable of being translated.
In this search I learned more or less of nine foreign languages, I read Oriental stuff in translations, I fought every University regulation and every professor who tried to make me learn anything except this, or who bothered me with "requirements for degrees".
Whilst at the university he met Hilda Doolittle, the daughter of the professor of astronomy. She developed as the poet known as H.D. She followed him to Europe in 1908, leaving her family, friends and country for little benefit to herself, and became involved with Pound in developing the Imagism movement in London. He sought her hand and in February that year asked her father, the astronomy professor Charles Doolittle, for his permission to marry. Doolittle was a curt man, described as "donnish" and intimidating. He was aware of Pound's reputation as a ladies' man, and unimpressed by his career as a poet, and constant moving. Doolittle's response was dismissive, he replied, "What! … Why you’re nothing but a nomad!" Pound asked Hilda to marry him in the summer of 1907, and though rejected, wrote several poems for her between 1905 and 1907. He hand-bound 25 of these, calling them Hilda's Book. He was seeing two other women at the same time – Viola Baxter and Mary Moore – later dedicating a book of poetry, Personae (1909), to the latter. He asked Mary to marry him that summer too, but she turned him down.
His parents and Frances Weston took him on another three-month European tour in 1902, after which he transferred, in 1903, to Hamilton College in Clinton, New York, possibly because of poor grades. Signed up for the Latin–Scientific course, he studied the Provençal dialect with William Pierce Shephard and Old English with Joseph D. Ibbotson; with Shephard he read Dante and from this began the idea for a long poem in three parts – of emotion, instruction and contemplation – planting the seeds for The Cantos. After graduating in 1905 with a PhB, he studied Romance languages under Hugo A. Rennert at the University of Pennsylvania, where he obtained an MA in the spring of 1906 and registered to write a PhD thesis on the jesters in Lope de Vega's plays. A Harrison fellowship covered his tuition fees and gave him a grant of $500, which he used to return to Europe.
Pound spent three weeks in Madrid in various libraries, including one in the royal palace. He happened to be standing outside the palace on 31 May 1906 during the attempted assassination by anarchists of King Alfonso, and left the country for fear he would be identified with them. After Spain he spent two weeks in Paris attending lectures at the Sorbonne, followed by a week in London. In July he returned to the United States, where in September his first essay, "Raphaelite Latin", was published in Book News Monthly. He took courses in the English department in 1907, where he annoyed Felix Schelling, the department head, with silly remarks during lectures, including that George Bernard Shaw was better than Shakespeare, and wound an enormous tin watch very slowly while Schelling spoke. As a result his fellowship was not renewed at the end of the year; Schelling told Pound that he was wasting his own time and that of the institution. Pound left without finishing his doctorate.
From the fall of 1907 Pound taught Romance languages at Wabash College in Crawfordsville, Indiana, a conservative town that he called the sixth circle of hell and an equally conservative college from which he was dismissed after deliberately provoking the college authorities. Smoking was forbidden, but he would smoke cigarillos in his office down the corridor from the president's. He annoyed his landlords by entertaining friends, including women, and was forced out of one house after two "stewdents found me sharing my meagre repast with the lady–gent impersonator in my privut apartments," as he told a friend. He was eventually caught in flagrante, although the details remain unclear and he denied any wrongdoing. The incident involved a stranded chorus girl to whom he offered tea and his bed for the night when she was caught in a snowstorm; when she was discovered the next morning by the landladies, his insistence that he had slept on the floor was met with disbelief and he was asked to leave the college. Glad to be free of the place, he left for Europe soon after, sailing from New York in March 1908.