Answer
$\left[\begin{array}{ccc}\frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} & 2 \sqrt{2}+3 \\ \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} & 7-5 \sqrt{2} \\ 0 & 0 & 1\end{array}\right]$
Work Step by Step
\[
\begin{array}{l}
{\left[\begin{array}{l}
x \\
y \\
1
\end{array}\right] \rightarrow\left[\begin{array}{lll}
1 & 0 & -3 \\
0 & 1 & -7 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{l}
x \\
y \\
1
\end{array}\right]} \\
{\left[\begin{array}{l}
x \\
y \\
1
\end{array}\right] \rightarrow\left[\begin{array}{lll}
\frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} & 0 \\
\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} & 0 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{lll}
1 & 0 & -3 \\
0 & 1 & -7 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{l}
x \\
y \\
1
\end{array}\right]} \\
{\left[\begin{array}{l}
x \\
y \\
1
\end{array}\right] \rightarrow\left[\begin{array}{lll}
1 & 0 & 3 \\
0 & 1 & 7 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{lll}
\frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} & 0 \\
\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} & 0 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{lll}
1 & 0 & -3 \\
0 & 1 & -7 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{l}
x \\
y \\
1
\end{array}\right]}
\end{array}
\]
rotation of points accomplished by first translating the figure by $-P$, then rotate about the origin, then again translate back by P.
\[
\left[\begin{array}{ccc}
1 & 0 & 3 \\
0 & 1 & 7 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{ccc}
\frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} & 0 \\
\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} & 0 \\
0 & 0 & 1
\end{array}\right]\left[\begin{array}{ccc}
1 & 0 & -3 \\
0 & 1 & -7 \\
0 & 0 & 1
\end{array}\right]=\left[\begin{array}{ccc}
\frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} & 2 \sqrt{2}+3 \\
\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} & 7-5 \sqrt{2} \\
0 & 0 & 1
\end{array}\right]
\]
