Answer
$\frac{(y')^2}{1}-\frac{(x')^2}{3}=1$
Work Step by Step
Step 1. Using the transformation formulas
$x=x'cos\theta-y'sin\theta$
$y=x'sin\theta+y'cos\theta$
we have
$x=x'cos45^\circ-y'sin45^\circ=\frac{\sqrt 2}{2}(x'-y')$
$y=x'sin45^\circ+y'cos45^\circ=\frac{\sqrt 2}{2}(x'+y')$
Step 2. The original equation becomes:
$x^2-4xy+y^2-3=(\frac{\sqrt 2}{2}(x'-y'))^2-4(\frac{\sqrt 2}{2}(x'-y'))(\frac{\sqrt 2}{2}(x'+y'))+(\frac{\sqrt 2}{2}(x'+y'))^2-3=\frac{1}{2}((x')^2-2x'y'+(y')^2)-2((x')^2-(y')^2)+\frac{1}{2}((x')^2+2x'y'+(y')^2)-3=-(x')^2+3(y')^2-3=0$
Step 3. We can rewrite the above result as $\frac{(y')^2}{1}-\frac{(x')^2}{3}=1$, which represents a hyperbola.
