Answer
$$[x]_{B'}=\left[ \begin {array}{ccc} 2\\ 1\\-1\\3 \end {array} \right].$$
Work Step by Step
Writing $x$ as a linear combination of the basis $B'$ as follows
$$x=\left(5,3,-6,2\right)= a\left(1,-1,2,1\right)+b\left(1,1,-4,3\right)+c\left(1,2,0,3\right)+d(1,2,-2,0).$$
We get the system
\begin{align*}
a+b+c+d&=5\\
-a+b+2c+2d&=3\\
2a-4b-2d&=-6\\
a+3b+3c&=2.
\end{align*}
By solving the above system we have the soluiton
$$a=2, \quad b= 1, \quad c=-1, \quad d=3.$$
Thus, the coordinate matrix of $x$ in $R^3$ relative to the
basis $B'$ is
$$[x]_{B'}=\left[ \begin {array}{ccc} 2\\ 1\\-1\\3 \end {array} \right].$$
