Answer
$R=\begin{bmatrix}
6&12\\0&6
\end{bmatrix}$
Work Step by Step
$R=Q^TA$
$R=\begin{bmatrix}
5/6&1/6&-3/6&1/6\\-1/6&5/6&1/6&3/6
\end{bmatrix}\begin{bmatrix}
5&9\\1&7\\-3&-5\\1&5
\end{bmatrix}=\begin{bmatrix}
6&12\\0&6
\end{bmatrix}$