Answer
$x= \frac{\sqrt 2}{2}(x'-y')$ and $y= \frac{\sqrt 2}{2}(x'+y')$
Work Step by Step
1. Based on the given equation, we have $A=3, B=-10, C=3$, the rotation angle satisfies $cot(2\theta)=\frac{A-C}{B}=0$, thus $2\theta=\frac{\pi}{2}$ and $\theta=\frac{\pi}{4}$.
2. The formulas for the rotation are $x=x'cos\theta-y'sin\theta=\frac{\sqrt 2}{2}(x'-y')$ and $y=x'sin\theta+y'cos\theta=\frac{\sqrt 2}{2}(x'+y')$
