Answer
$$\frac{{{x^3}{{\ln }^2}x}}{3} - \frac{{2{x^3}\ln x}}{9} + \frac{2}{{27}}{x^3} + C$$
Work Step by Step
$$\eqalign{
& \int {{x^2}{{\ln }^2}x} dx \cr
& {\text{substitute }}u = {\left( {\ln x} \right)^2},{\text{ }}du = 2\left( {\ln x} \right)\left( {\frac{1}{x}} \right)dx,{\text{ }}du = \frac{{2\ln x}}{x}dx \cr
& dv = {x^2}dx,{\text{ }}v = \frac{{{x^3}}}{3} \cr
& {\text{applying integration by parts}}{\text{, we have}} \cr
& \int {{x^2}{{\ln }^2}x} dx = \frac{{{x^3}{{\left( {\ln x} \right)}^2}}}{3} - \int {\left( {\frac{{{x^3}}}{3}} \right)\left( {\frac{{2\ln x}}{x}} \right)} dx \cr
& \int {{x^2}{{\ln }^2}x} dx = \frac{{{x^3}{{\ln }^2}x}}{3} - \frac{2}{3}\int {{x^2}\ln x} dx \cr
& {\text{substitute }}u = \ln x,{\text{ }}du = \frac{1}{x}dx \cr
& dv = {x^2}dx,{\text{ }}v = \frac{{{x^3}}}{3} \cr
& {\text{applying integration by parts}}{\text{, we have}} \cr
& \int {{x^2}{{\ln }^2}x} dx = \frac{{{x^3}{{\ln }^2}x}}{3} - \frac{2}{3}\left( {\frac{{{x^3}\ln x}}{3} - \int {\frac{{{x^3}}}{3}\frac{1}{x}dx} } \right) \cr
& \int {{x^2}{{\ln }^2}x} dx = \frac{{{x^3}{{\ln }^2}x}}{3} - \frac{2}{3}\left( {\frac{{{x^3}\ln x}}{3} - \frac{1}{3}\int {{x^2}dx} } \right) \cr
& \int {{x^2}{{\ln }^2}x} dx = \frac{{{x^3}{{\ln }^2}x}}{3} - \frac{{2{x^3}\ln x}}{9} + \frac{2}{9}\int {{x^2}dx} \cr
& {\text{integrate}} \cr
& \int {{x^2}{{\ln }^2}x} dx = \frac{{{x^3}{{\ln }^2}x}}{3} - \frac{{2{x^3}\ln x}}{9} + \frac{2}{9}\left( {\frac{{{x^3}}}{3}} \right) + C \cr
& \int {{x^2}{{\ln }^2}x} dx = \frac{{{x^3}{{\ln }^2}x}}{3} - \frac{{2{x^3}\ln x}}{9} + \frac{2}{{27}}{x^3} + C \cr} $$
