Answer
$x= \frac{1}{5}(3x'-4y')$ and $y= \frac{1}{5}(4x'+3y')$
Work Step by Step
Step 1. From the given equation, we have $A=10, B=24, C=17$. Thus we have $cot2\theta=\frac{A-C}{B}=\frac{10-17}{24}=-\frac{7}{24}$ Thus $2\theta$ is in Quadrant II and $cos2\theta=-\frac{7}{\sqrt {7^2+24^2}}=-\frac{7}{25}$ (form a right triangle with sides $7,24,25$ here).
Step 2. We can find values of $sin\theta$ and $\cos\theta$ as $sin\theta=\sqrt {\frac{1-cos2\theta}{2}}=\sqrt {\frac{1-(-\frac{7}{25})}{2}}=\frac{4}{5}$ and $cos\theta=\sqrt {1-sin^2\theta}=\frac{3}{5}$
Step 3. Using the transformation formula $x=x'cos\theta-y'sin\theta$ and $y=x'sin\theta+y'cos\theta$, we have
$x=x'(\frac{3}{5})-y'(\frac{4}{5})=\frac{1}{5}(3x'-4y')$
$y=x'(\frac{4}{5})+y'(\frac{3}{5})=\frac{1}{5}(4x'+3y')$
