Answer
$$\frac{{{x^2}\sin 5x}}{5} + \frac{{2x\cos 5x}}{{25}} - \frac{{2\sin 5x}}{{125}} + C$$
Work Step by Step
$$\eqalign{ & \int {{x^2}\cos 5x} dx \cr & {\text{Using the reduction formula }} \cr & \int {{x^n}\cos axdx} = \frac{{{x^n}\sin ax}}{a} - \frac{n}{a}\int {{x^{n - 1}}\sin axdx} ,\,\,\,\,\,{\text{for }}a \ne 0 \cr & {\text{then,}} \cr & \int {{x^2}\cos 5x} dx = \frac{{{x^2}\sin 5x}}{5} - \frac{2}{5}\int {{x^{2 - 1}}\sin 5xdx} \cr & \int {{x^2}\cos 5x} dx = \frac{{{x^2}\sin 5x}}{5} - \frac{2}{5}\int {x\sin 5xdx} \cr & {\text{Using the reduction formula }} \cr & \int {{x^n}\sin axdx} = - \frac{{{x^n}\cos ax}}{a} + \frac{n}{a}\int {{x^{n - 1}}\cos axdx} ,\,\,\,\,\,{\text{for }}a \ne 0 \cr & \int {{x^2}\cos 5x} dx = \frac{{{x^2}\sin 5x}}{5} - \frac{2}{5}\left( { - \frac{{x\cos 5x}}{5} + \frac{1}{5}\int {{x^{1 - 1}}\cos 5xdx} } \right) \cr & \int {{x^2}\cos 5x} dx = \frac{{{x^2}\sin 5x}}{5} - \frac{2}{5}\left( { - \frac{{x\cos 5x}}{5} + \frac{1}{5}\int {\cos 5xdx} } \right) \cr & \int {{x^2}\cos 5x} dx = \frac{{{x^2}\sin 5x}}{5} + \frac{{2x\cos 5x}}{{25}} - \frac{2}{{25}}\int {\cos 5xdx} \cr & \int {{x^2}\cos 5x} dx = \frac{{{x^2}\sin 5x}}{5} + \frac{{2x\cos 5x}}{{25}} - \frac{2}{{25}}\left( {\frac{1}{5}\sin 5x} \right) + C \cr & \int {{x^2}\cos 5x} dx = \frac{{{x^2}\sin 5x}}{5} + \frac{{2x\cos 5x}}{{25}} - \frac{{2\sin 5x}}{{125}} + C \cr} $$