Answer
(a) Ellipse.
(b) $\phi=\frac{\pi}{4}$, $\frac{X^2}{6}+\frac{2Y^2}{3}=1$
(c) See graph.
Work Step by Step
(a) Write the equation as $5x^2-6xy+5y^2-8\sqrt 2x+8\sqrt 2y-4=0$, we can identify $A=5, B=-6, C=5$. The discriminant is $B^2-4AC=36-100=-64$, thus the graph of the equation will be an ellipse.
(b) Step 1. To eliminate the xy-term, we need to rotate the axes with an angle $\phi$ given by $cot2\phi=\frac{A-C}{B}=0$ which gives $2\phi=\frac{\pi}{2}$ and $\phi=\frac{\pi}{4}$
Step 2. Use the conversion formulas $x=Xcos\phi-Ysin\phi=\frac{\sqrt 2}{2}(X-Y)$ and $y=Xsin\phi+Ycos\phi=\frac{\sqrt 2}{2}(X+Y)$
Step 3. Plug the above relation into the original equation to get $5(\frac{\sqrt 2}{2}(X-Y))^2-6(\frac{\sqrt 2}{2}(X-Y))(\frac{\sqrt 2}{2}(X+Y))+5(\frac{\sqrt 2}{2}(X+Y))^2-8\sqrt 2(\frac{\sqrt 2}{2}(X-Y))+8\sqrt 2(\frac{\sqrt 2}{2}(X+Y))-4=0$
Step 4. Multiply 2 to both sides and simplify the equation to get $4X^2+16Y^2+32Y-8=0$ or $X^2+4(Y^2+2Y+1)=2+4=6$ which gives $\frac{X^2}{6}+\frac{2Y^2}{3}=1$
(c) See graph.
