Answer
$$\frac{1}{5}{x^2}{e^{5x}} - \frac{2}{{25}}x{e^{5x}} + \frac{2}{{125}}{e^{5x}} + C$$
Work Step by Step
$$\eqalign{
& \int {{x^2}{e^{5x}}} dx \cr
& {\text{a matching integral in a table of integrals at the end of the book is the }} \cr
& {\text{reduction formula 102}} \cr
& \int {{x^n}{e^{ax}}dx} = \frac{1}{a}{x^n}{e^{ax}} - \frac{n}{a}\int {{x^{n - 1}}{e^{ax}}dx} \cr
& n = 2,{\text{ }}a = 5 \cr
& \int {{x^2}{e^{5x}}} dx = \frac{1}{5}{x^2}{e^{5x}} - \frac{2}{5}\int {{x^{2 - 1}}{e^{5x}}dx} \cr
& \int {{x^2}{e^{5x}}} dx = \frac{1}{5}{x^2}{e^{5x}} - \frac{2}{5}\int {x{e^{5x}}dx} \cr
& n = 1,{\text{ }}a = 5 \cr
& \int {{x^2}{e^{5x}}} dx = \frac{1}{5}{x^2}{e^{5x}} - \frac{2}{5}\left( {\frac{1}{5}x{e^{5x}} - \frac{1}{5}\int {{x^{1 - 1}}{e^{5x}}dx} } \right) \cr
& \int {{x^2}{e^{5x}}} dx = \frac{1}{5}{x^2}{e^{5x}} - \frac{2}{5}\left( {\frac{1}{5}x{e^{5x}} - \frac{1}{5}\int {{e^{5x}}dx} } \right) \cr
& \int {{x^2}{e^{5x}}} dx = \frac{1}{5}{x^2}{e^{5x}} - \frac{2}{{25}}x{e^{5x}} + \frac{2}{{25}}\int {{e^{5x}}dx} \cr
& {\text{formula 9 }}\int {{e^{ax}}dx = \frac{1}{a}{e^{ax}} + C} \cr
& \int {{x^2}{e^{5x}}} dx = \frac{1}{5}{x^2}{e^{5x}} - \frac{2}{{25}}x{e^{5x}} + \frac{2}{{25}}\left( {\frac{1}{5}{e^{5x}}} \right) + C \cr
& \int {{x^2}{e^{5x}}} dx = \frac{1}{5}{x^2}{e^{5x}} - \frac{2}{{25}}x{e^{5x}} + \frac{2}{{125}}{e^{5x}} + C \cr} $$
