Answer
$\frac{(x')^2}{4}+\frac{(y')^2}{1}=1$
Work Step by Step
Step 1. Using the transformation formulas
$x=x'cos\theta-y'sin\theta$
$y=x'sin\theta+y'cos\theta$
we have
$x=x'cos60^\circ-y'sin60^\circ=\frac{1}{2}(x'-\sqrt 3y')$
$y=x'sin60^\circ+y'cos60^\circ=\frac{1}{2}(\sqrt 3x'+y')$
Step 2. The original equation becomes:
$13x^2-6\sqrt 3xy+7y^2-16=13(\frac{1}{2}(x'-\sqrt 3y'))^2-6\sqrt 3(\frac{1}{2}(x'-\sqrt 3y'))(\frac{1}{2}(\sqrt 3x'+y'))+7(\frac{1}{2}(\sqrt 3x'+y'))^2-16=\frac{13}{4}((x')^2-2\sqrt 3x'y'+3(y')^2)-\frac{6\sqrt 3}{4}(\sqrt 3(x')^2-2x'y'-\sqrt 3(y')^2)+\frac{7}{4}(3(x')^2+2\sqrt 3x'y'+(y')^2)-16=4(x')^2+16(y')^2-16=0$
Step 3. We can rewrite the above result as $\frac{(x')^2}{4}+\frac{(y')^2}{1}=1$, which represents an ellipse.
