Answer
$$\int^{\pi/3}_0 x\tan^2 xdx=\frac{\pi\sqrt3}{3}-\frac{\pi^2}{18}+\ln\frac{1}{2}$$
Work Step by Step
$$A=\int^{\pi/3}_0 x\tan^2 xdx$$
Here, $\tan^2x$ makes any attempts to take integral or derivative become difficult. So we need to substitute it with something else.
We have $$\tan^2x=\frac{\sin^2x}{\cos^2x}=\frac{1-\cos^2x}{\cos^2x}=\frac{1}{\cos^2x}-1=\sec^2x-1$$
Therefore, $$A=\int^{\pi/3}_0 x(\sec^2x-1)dx$$
Take $u=x$ and $dv=(\sec^2x-1)dx$
Then $du=dx$ and $v=\tan x-x$
Apply the formula $\int^b_audv=uv]^b_a-\int^b_avdu$, we have
$$A=x(\tan x-x)\Big]^{\pi/3}_0-\int^{\pi/3}_0(\tan x-x)dx$$
Here we need to learn $\int \tan xdx$ first. We have $$\int\tan xdx=\int\frac{\sin x}{\cos x}dx=\int\frac{-1}{\cos x}d(\cos x)=-\ln|\cos x|+C$$
Return back to $A$, we have
$$A=\frac{\pi}{3}\Big(\tan\frac{\pi}{3}-\frac{\pi}{3}\Big)-0-\Big(-\ln|\cos x|-\frac{x^2}{2}\Big)\Bigg]^{\pi/3}_0$$ $$A=\frac{\pi}{3}\Big(\tan\frac{\pi}{3}-\frac{\pi}{3}\Big)+\Big(\ln|\cos x|+\frac{x^2}{2}\Big)\Bigg]^{\pi/3}_0$$ $$A=\frac{\pi}{3}\Big(\tan\frac{\pi}{3}-\frac{\pi}{3}\Big)+\Big(\ln|\cos\frac{\pi}{3}|+\frac{\pi^2}{18}\Big)-\Big(\ln|\cos0|+0\Big)$$ $$A=\frac{\pi}{3}\Big(\sqrt3-\frac{\pi}{3}\Big)+\Big(\ln\frac{1}{2}+\frac{\pi^2}{18}\Big)-\ln1$$ $$A=\frac{\pi\sqrt3}{3}-\frac{\pi^2}{9}+\frac{\pi^2}{18}+\ln\frac{1}{2}$$ $$A=\frac{\pi\sqrt3}{3}-\frac{\pi^2}{18}+\ln\frac{1}{2}$$
