Answer
$X^2-2\sqrt 3 XY+3Y^2+(2\sqrt 3-4)X+(4\sqrt 3-2)Y+4=0$
Work Step by Step
Step 1. Identify the given quantities: equation $y=(x-1)^2$, rotation $\phi=60^{\circ}$
Step 2. Recall the formula for the transformation: $x=X\cdot cos60^{\circ} - Y\cdot sin60^{\circ}=\frac{1}{2}X-\frac{\sqrt 3}{2}Y$ and $y=-X\cdot sin60^{\circ}+Y\cdot cos60^{\circ}=-\frac{\sqrt 3}{2}X+\frac{1}{2}Y$
Step 3. Plug-in the expressions above to the original equation, we get $-\frac{\sqrt 3}{2}X+\frac{1}{2}Y=(\frac{1}{2}X-\frac{\sqrt 3}{2}Y-1)^2$
Step 4. Multiply 4 to both sides of the above equation and simply the right side:
$-2\sqrt 3 X+2Y=( X-\sqrt 3Y-2)^2=(X-\sqrt 3Y)^2-4(X-\sqrt 3Y)+4=X^2-2\sqrt 3 XY+3Y^2-4X+4\sqrt 3Y+4$
Step 5. Combine like terms of both side to get $X^2-2\sqrt 3 XY+3Y^2+(2\sqrt 3-4)X+(4\sqrt 3-2)Y+4=0$