Answer
$-\frac{\pi}{6}$
Work Step by Step
Because $\tan(\pi-x)=-\tan x$ and $-\tan x=\tan(-x)$, we can write:
$\tan \frac{5\pi}{6}=\tan(\pi-\frac{\pi}{6})=-\tan\frac{\pi}{6}=\tan(-\frac{\pi}{6})$.
So, $\tan^{-1}(\tan\frac{5\pi}{6})=\tan^{-1}(\tan -\frac{\pi}{6})$
Within the restricted interval ($-\frac{\pi}{2}\lt x\lt\frac{\pi}{2}$),
$\tan^{-1}(\tan x)=x$
$\implies \tan^{-1}(\tan \frac{5\pi}{6})=\tan^{-1}(\tan -\frac{\pi}{6})=-\frac{\pi}{6}$
