Answer
(a) hyperbola.
(b) $\phi\approx53.13^{\circ}$.
$\frac{X^2}{4}-Y^2=1$
(c) See graph.
Work Step by Step
(a) Rewrite the original equation as $11x^2-24xy+4y^2+0x+0y+20=0$ and we have $A=11, B=-24, C=4, D=E=0,F=20$. The discriminant is $B^2-4AC=24^2-176=400\gt0$, thus the graph of the equation is a hyperbola.
(b) To eliminate the xy-term, we need a rotation of axes with an angle $\phi$ where $cot2\phi=\frac{A-C}{B}=\frac{11-4}{-24}=-\frac{7}{24}$ which gives $2\phi\approx106.26^{\circ}$ and $\phi\approx53.13^{\circ}$.
With $tan2\phi==-\frac{24}{7}$, we set $2\phi$ in Quadrant II and get $cos2\phi=-\frac{7}{25}$. Use the half angle formula with $\phi$ in Quadrant I to get $sin\phi=\sqrt {\frac{1-cos2\phi}{2}}=\frac{4}{5}$ and $cos\phi=\sqrt {\frac{1+cos2\phi}{2}}=\frac{3}{5}$
The transformation formula gives:
$x=X\cdot\frac{3}{5} - Y\cdot \frac{4}{5}=\frac{1}{5}(3X-4Y) $,
$y=X\cdot \frac{4}{5}+Y\cdot\frac{3}{5}=\frac{1}{5}(4X+3Y) $
Use them in the original equation to get $11(\frac{1}{5}(3X-4Y))^2-24(\frac{1}{5}(3X-4Y))(\frac{1}{5}(4X+3Y))+4(\frac{1}{5}(4X+3Y))^2+20=0$
Simplify this equation by multiplying 25 and combine like terms to get $-125X^2+500Y^2+500=0$ or $\frac{X^2}{4}-Y^2=1$
(c) See graph.
