Answer
c)
Work Step by Step
To convert from polar to rectangular coordinates we use $x=r\cos\theta$ and $y=r\sin\theta$. To convert from rectangular to polar coordinates, we use $r=\pm\sqrt{x^2+y^2}$, $\tan\theta=\frac{y}{x}$, but aware of which quadrant the point lies in.
Hence here $x^2+y^2+xy=1\\r^2+r\cos\theta r\sin\theta=1\\r^2(1+\cos\theta\sin\theta)=1\\r^2=\frac{1}{1+\cos\theta\sin\theta}\\r=\frac{1}{\sqrt{1+\cos\theta\sin\theta}}$.
Thus the answer is c).