Answer
$$\int^2_{2/\sqrt3} t\sec^{-1}tdt=\frac{5\pi}{9}-\frac{\sqrt3}{3}$$
Work Step by Step
$$A=\int^2_{2/\sqrt3} t\sec^{-1}tdt$$
Take $u=\sec^{-1}t$ and $dv=tdt$
Then $du=\frac{1}{t\sqrt{t^2-1}}$ and $v=\frac{t^2}{2}$
Apply the formula $\int^b_audv=uv]^b_a-\int^b_avdu$
$$A=\Big(\frac{t^2}{2}\sec^{-1}t\Big)\Big]^2_{2/\sqrt3}-\int^2_{2/\sqrt3}\frac{t^2}{2}\times\frac{1}{t\sqrt{t^2-1}}dt$$ $$A=\Big(2\sec^{-1}2-\frac{\frac{4}{3}}{2}\sec^{-1}\frac{2}{\sqrt3}\Big)-\frac{1}{2}\int^2_{2/\sqrt3}\frac{t}{\sqrt{t^2-1}}dt$$ $$A=\Big(2\times\frac{\pi}{3}-\frac{2}{3}\times\frac{\pi}{6}\Big)-\frac{1}{2}\int^2_{2/\sqrt3}\frac{t}{\sqrt{t^2-1}}dt$$ $$A=\frac{2\pi}{3}-\frac{\pi}{9}-\frac{1}{2}\int^2_{2/\sqrt3}\frac{t}{\sqrt{t^2-1}}dt$$ $$A=\frac{5\pi}{9}-\frac{1}{2}\int^2_{2/\sqrt3}\frac{t}{\sqrt{t^2-1}}dt$$
Let $a=\sqrt{t^2-1}$. Then $da=\frac{(t^2-1)'}{2\sqrt{t^2-1}}dt=\frac{2t}{2a}dt=\frac{t}{a}dt$
Therefore, $tdt=ada$
Also, for $t=2$, $a=\sqrt3$ and for $t=2/\sqrt3$, $a=\sqrt{\frac{4}{3}-1}=\sqrt{\frac{1}{3}}$
$$A=\frac{5\pi}{9}-\frac{1}{2}\int^\sqrt3_{\sqrt{1/3}}\frac{a}{a}da$$ $$A=\frac{5\pi}{9}-\frac{1}{2}\int^{\sqrt3}_\sqrt{1/3}da$$ $$A=\frac{5\pi}{9}-\frac{1}{2}\Big(a\Big]^{\sqrt3}_\sqrt{1/3}\Big)$$ $$A=\frac{5\pi}{9}-\frac{1}{2}\Big(\sqrt3-\sqrt\frac{1}{3}\Big)$$ $$A=\frac{5\pi}{9}-\frac{1}{2}\Big(\sqrt3-\frac{\sqrt3}{3}\Big)=\frac{5\pi}{9}-\frac{1}{2}\times\frac{2\sqrt3}{3}$$ $$A=\frac{5\pi}{9}-\frac{\sqrt3}{3}$$
