Answer
(a) Ellipse.
(b) $\phi\approx26.6^{\circ}$, $\frac{X^2}{3}+\frac{Y^2}{18}=1$
(c) See graph.
(d) $V_1(-\frac{3\sqrt {10}}{5}, \frac{6\sqrt {10}}{5})$, $V_2(\frac{3\sqrt {10}}{5}, -\frac{6\sqrt {10}}{5})$.
Work Step by Step
(a) Rewrite the equation as $5x^2+4xy+2y^2-18=0$ and we can identify that $A=5, B=4, C=2$. The discriminant is $B^2-4AC=16-40=-26\lt0$, thus the graph of the equation will be an ellipse.
(b) Step 1. To eliminate the xy-term, we need to rotate the axes with an angle $\phi$ given by $cot2\phi=\frac{A-C}{B}=\frac{5-2}{4}=\frac{3}{4}$ which gives $2\phi\approx53.13^{\circ}$ and $\phi\approx26.6^{\circ}$
Step 2. With $tan2\phi=\frac{4}{3}$, we get $cos2\phi=\frac{3}{5}$ and $sin\phi=\sqrt {\frac{1-cos2\phi}{2}}=\frac{\sqrt 5}{5}$, $cos\phi=\sqrt {\frac{1+cos2\phi}{2}}=\frac{2\sqrt 5}{5}$
Step 3. Use the conversion formulas $x=Xcos\phi-Ysin\phi=\frac{\sqrt 5}{5}(2X-Y)$ and $y=Xsin\phi+Ycos\phi=\frac{\sqrt 5}{5}(X+2Y)$
Step 4. Plug the above relation into the original equation to get $5(\frac{\sqrt 5}{5}(2X-Y))^2+4(\frac{\sqrt 5}{5}(2X-Y))(\frac{\sqrt 5}{5}(X+2Y))+2(\frac{\sqrt 5}{5}(X+2Y))^2-18=0$
Step 5. Multiply 5 to both sides and simplify the equation to get $30X^2+5Y^2=18\times5$ or $\frac{X^2}{3}+\frac{Y^2}{18}=1$
(c) See graph.
(d) Based on the equation obtained in part (b), the vertices in the XY-system are $V(0, \pm3\sqrt 2)$. Use the conversion formula $x=Xcos\phi-Ysin\phi=\frac{\sqrt 5}{5}(2X-Y)$ and $y=Xsin\phi+Ycos\phi=\frac{\sqrt 5}{5}(X+2Y)$, the first vertex is given by $x=\frac{\sqrt 5}{5}(0-3\sqrt 2)=-\frac{3\sqrt {10}}{5}$ and $y=\frac{\sqrt 5}{5}(0+2(3\sqrt 2))=\frac{6\sqrt {10}}{5}$, thus we have $V_1(-\frac{3\sqrt {10}}{5}, \frac{6\sqrt {10}}{5})$.
Similarly, for the second vertex, $x=\frac{\sqrt 5}{5}(0+3\sqrt 2)=\frac{3\sqrt {10}}{5}$ and $y=\frac{\sqrt 5}{5}(0-2(3\sqrt 2))=-\frac{6\sqrt {10}}{5}$, thus we have $V_2(\frac{3\sqrt {10}}{5}, -\frac{6\sqrt {10}}{5})$.