Answer
$\frac{1}{2}{x^2}{\ln ^2}x - \frac{1}{2}{x^2}\ln x + \frac{1}{4}{x^2} + C$
Work Step by Step
$$\eqalign{
& \int x {\left( {\ln x} \right)^2}dx \cr
& {\text{Integrating by parts}}{\text{, }} \cr
& {\text{Let }}u = {\left( {\ln x} \right)^2},{\text{ }}du = 2\left( {\ln x} \right)\left( {\frac{1}{x}} \right)dx \cr
& {\text{ }}dv = xdx,{\text{ }}v = \frac{1}{2}{x^2} \cr
& {\text{Using integration by parts formula}} \cr
& \int u dv = uv - \int {vdu} \cr
& \int x {\left( {\ln x} \right)^2}dx = {\left( {\ln x} \right)^2}\left( {\frac{1}{2}{x^2}} \right) - \int {\left( {\frac{1}{2}{x^2}} \right)\left( 2 \right)} \left( {\ln x} \right)\left( {\frac{1}{x}} \right)dx \cr
& {\text{Multiply}} \cr
& \int x {\left( {\ln x} \right)^2}dx = \frac{1}{2}{x^2}{\ln ^2}x - \int {x\ln x} dx \cr
& {\text{Integrating by parts again }} \cr
& {\text{Let }}u = \ln x,{\text{ }}du = \frac{1}{x}dx \cr
& {\text{ }}dv = xdx,{\text{ }}v = \frac{1}{2}{x^2} \cr
& \int x {\left( {\ln x} \right)^2}dx = \frac{1}{2}{x^2}{\ln ^2}x - \left( {\frac{1}{2}{x^2}\ln x - \int {\left( {\frac{1}{2}{x^2}} \right)\left( {\frac{1}{x}} \right)dx} } \right) \cr
& \int x {\left( {\ln x} \right)^2}dx = \frac{1}{2}{x^2}{\ln ^2}x - \frac{1}{2}{x^2}\ln x + \frac{1}{2}\int {xdx} \cr
& {\text{Integrate}} \cr
& \int x {\left( {\ln x} \right)^2}dx = \frac{1}{2}{x^2}{\ln ^2}x - \frac{1}{2}{x^2}\ln x + \frac{1}{2}\left( {\frac{{{x^2}}}{2}} \right) + C \cr
& \int x {\left( {\ln x} \right)^2}dx = \frac{1}{2}{x^2}{\ln ^2}x - \frac{1}{2}{x^2}\ln x + \frac{1}{4}{x^2} + C \cr} $$
