Answer
$$\left\{ {\left( {\sqrt 2 ,\sqrt 2 } \right),\left( { - \sqrt 2 , - \sqrt 2 } \right),\left( {\frac{{2\sqrt 5 }}{5}i, - \sqrt 5 i} \right),\left( { - \frac{{2\sqrt 5 }}{5}i,\sqrt 5 i} \right)} \right\}$$
Work Step by Step
$$\eqalign{
& \,\,5{x^2} - 2{y^2} = 6\,\,\,\,\left( {\bf{1}} \right) \cr
& \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,xy = 2\,\,\,\left( {\bf{2}} \right) \cr
& \cr
& {\text{Solve the equation }}\left( {\bf{2}} \right){\text{ for }}y \cr
& xy = 12 \cr
& y = \frac{2}{x} \cr
& \cr
& {\text{Substitute }}\frac{2}{x}{\text{ for }}y{\text{ into the equation }}\left( {\bf{1}} \right) \cr
& 5{x^2} - 2{\left( {\frac{2}{x}} \right)^2} = 6 \cr
& {\text{Solve for }}x \cr
& 5{x^2} - \frac{8}{{{x^2}}} = 6 \cr
& 5{x^4} - 8 = 6{x^2} \cr
& 5{x^4} - 6{x^2} - 8 = 0 \cr
& \left( {{x^2} - 2} \right)\left( {5{x^2} + 4} \right) = 0 \cr
& {x_1} = \sqrt 2 ,\,\,\,\,{x_2} = - \sqrt 2 ,\,\,\,\,{x_3} = \frac{{2\sqrt 5 }}{5}i,\,\,\,\,{x_4} = - \frac{{2\sqrt 5 }}{5}i \cr
& \cr
& {\text{Substitute }}{x_1} = \sqrt 2 {\text{ into the equation }}y = \frac{2}{x} \cr
& y = \frac{2}{{\sqrt 2 }} \cr
& y = \sqrt 2 \cr
& {\text{The first solution is }}\left( {\sqrt 2 ,\sqrt 2 } \right) \cr
& \cr
& {\text{Substitute }}{x_2} = - \sqrt 2 {\text{ into the equation }}y = \frac{2}{x} \cr
& y = \frac{2}{{ - \sqrt 2 }} \cr
& y = - \sqrt 2 \cr
& {\text{The second solution is }}\left( { - \sqrt 2 , - \sqrt 2 } \right) \cr
& \cr
& {\text{Substitute }}{x_3} = \frac{{2\sqrt 5 }}{5}i{\text{ into the equation }}y = \frac{2}{x} \cr
& y = \frac{2}{{\frac{{2\sqrt 5 }}{5}i}} \cr
& y = - \sqrt 5 i \cr
& {\text{The third solution is }}\left( {\frac{{2\sqrt 5 }}{5}i, - \sqrt 5 i} \right) \cr
& \cr
& {\text{Substitute }}{x_4} = - \frac{{2\sqrt 5 }}{5}i{\text{ into the equation }}y = \frac{2}{x} \cr
& y = \frac{2}{{ - \frac{{2\sqrt 5 }}{5}i}} \cr
& y = \sqrt 5 i \cr
& {\text{The fourth solution is }}\left( { - \frac{{2\sqrt 5 }}{5}i,\sqrt 5 i} \right) \cr
& \cr
& {\text{Therefore, the solution set of the system is}} \cr
& \left\{ {\left( {\sqrt 2 ,\sqrt 2 } \right),\left( { - \sqrt 2 , - \sqrt 2 } \right),\left( {\frac{{2\sqrt 5 }}{5}i, - \sqrt 5 i} \right),\left( { - \frac{{2\sqrt 5 }}{5}i,\sqrt 5 i} \right)} \right\} \cr} $$
