Answer
$$\left\{ {\left( {3,4} \right),\left( {3, - 4} \right),\left( {\frac{{4\sqrt 3 }}{3}i, - 3i\sqrt 3 } \right),\left( { - \frac{{4\sqrt 3 }}{3}i,3\sqrt 3 i} \right)} \right\}$$
Work Step by Step
$$\eqalign{
& \,\,3{x^2} - {y^2} = 11\,\,\,\,\,\left( {\bf{1}} \right) \cr
& \,\,\,\,\,\,\,\,\,\,\,\,\,\,xy = 12\,\,\,\left( {\bf{2}} \right) \cr
& \cr
& {\text{Solve the equation }}\left( {\bf{2}} \right){\text{ for }}y \cr
& xy = 12 \cr
& y = \frac{{12}}{x} \cr
& \cr
& {\text{Substitute }}\frac{{12}}{x}{\text{ for }}y{\text{ into the equation }}\left( {\bf{1}} \right) \cr
& 3{x^2} - {\left( {\frac{{12}}{x}} \right)^2} = 11 \cr
& {\text{Solve for }}x \cr
& 3{x^2} - \frac{{144}}{{{x^2}}} = 11 \cr
& 3{x^4} - 144 = 11{x^2} \cr
& 3{x^4} - 11{x^2} - 144 = 0 \cr
& \left( {{x^2} - 9} \right)\left( {3{x^2} + 16} \right) = 0 \cr
& {x_1} = 3,\,\,\,\,{x_2} = - 3,\,\,\,\,{x_3} = \frac{{4\sqrt 3 }}{3}i,\,\,\,\,{x_4} = - \frac{{4\sqrt 3 }}{3}i \cr
& \cr
& {\text{Substitute }}{x_1} = 3{\text{ into the equation }}y = \frac{{12}}{x} \cr
& y = \frac{{12}}{3} \cr
& y = 4 \cr
& {\text{The first solution is }}\left( {3,4} \right) \cr
& \cr
& {\text{Substitute }}{x_2} = - 3{\text{ into the equation }}y = \frac{{12}}{x} \cr
& y = \frac{{12}}{{ - 3}} \cr
& y = - 4 \cr
& {\text{The second solution is }}\left( {3, - 4} \right) \cr
& \cr
& {\text{Substitute }}{x_3} = - \frac{{4\sqrt 3 }}{3}i{\text{ into the equation }}y = \frac{{12}}{x} \cr
& y = \frac{{12}}{{\frac{{4\sqrt 3 }}{3}i}} \cr
& y = - 3i\sqrt 3 \cr
& {\text{The third solution is }}\left( {\frac{{4\sqrt 3 }}{3}i, - 3i\sqrt 3 } \right) \cr
& \cr
& {\text{Substitute }}{x_4} = - \frac{{4\sqrt 3 }}{3}i{\text{ into the equation }}y = \frac{{12}}{x} \cr
& y = \frac{2}{{ - \frac{{4\sqrt 3 }}{3}i}} \cr
& y = 3\sqrt 3 i \cr
& {\text{The second solution is }}\left( { - \frac{{4\sqrt 3 }}{3}i,3\sqrt 3 i} \right) \cr
& \cr
& {\text{Therefore, the solution set of the system is}} \cr
& \left\{ {\left( {3,4} \right),\left( {3, - 4} \right),\left( {\frac{{4\sqrt 3 }}{3}i, - 3i\sqrt 3 } \right),\left( { - \frac{{4\sqrt 3 }}{3}i,3\sqrt 3 i} \right)} \right\} \cr} $$
