Answer
$x= \frac{\sqrt {13}}{13}(3x'-2y')$ and $y= \frac{\sqrt {13}}{13}(2x'+3y')$
Work Step by Step
1. Based on the given equation, we have $A=25, B=-36, C=40$, the rotation angle satisfies $cot(2\theta)=\frac{A-C}{B}=\frac{25-40}{-36}=\frac{5}{12}$ (quadrant I),
2. Let $x=5, y=12$, we have $r=13$, $cos(2\theta)=\frac{5}{13}$, with $\theta$ in quadrant I, thus $sin\theta=\sqrt {\frac{1-cos(2\theta)}{2}}=\sqrt {\frac{1-\frac{5}{13}}{2}}=\frac{2\sqrt {13}}{13}$ and
$cos\theta=\sqrt {\frac{1+cos(2\theta)}{2}}=\sqrt {\frac{1+\frac{5}{13}}{2}}=\frac{3\sqrt {13}}{13}$
3. The formulas for the rotation are $x=x'cos\theta-y'sin\theta=\frac{3\sqrt {13}}{13}x'-\frac{2\sqrt {13}}{13}y'=\frac{\sqrt {13}}{13}(3x'-2y')$ and $y=x'sin\theta+y'cos\theta=\frac{2\sqrt {13}}{13}x'+\frac{3\sqrt {13}}{13}y'=\frac{\sqrt {13}}{13}(2x'+3y')$
