Answer
$f(x) =x log x$ is $O(x^{2})$ with $k=1$ and $C=1$.
$f(x)=x^{2}$ is not $O(x log x)$.
Work Step by Step
$Frist$ $part$
$f(x) =x log x$
$g(x) =x^{2}$
Property of logarithms: $log x<=x$ when $x>=0$.
When $x>1$, then the logarithm is positive $log>0$.
For convenience sake, we will then choose $k=1$ and thus use $x>1$.
(Note: you could choose a different value of $k$, which will lead to a different value for $C$).
$|f(x)|=|x \log x|$
$\quad=x \log x$
$\leq x \cdot x$
$=x^{2}$
$=\left|x^{2}\right|$
Thus we need to choose $C$ at least $1$.Let us then take $C=1$.
By the definition of Big-O notation, $f(x) =x log x$ is $O(x^{2})$ with $k=1$ and $C=1$.
$Second$ $part$
$f(x) =x^{2}$
$g(x) =x log x$
Let us assume that $f(x) =x^{2}$ is $O(x log x)$
(Note: if we obtain a contradiction then we know that this statement is not true).
Then there exist constants $C$ and $k$ such that:
$\left|x^{2}\right| \leq C|x \log x|$
When $x>k$.
Let us assume $C>=1$, which is a safe assumption (because if the statement is true for a value of $C$ between $0$ and $1$, then it is also true for any value larger than $1$).
$\left|x^{2}\right| \leq C|x \log x|=|C x \log x|$
We have then obtained a contradiction, because $|x|>|\log x|$ when $x>0$ and thus also $\left|x^{2}\right|>|C x \log x|$.
Thus $f(x)=x^{2}$ is not $O(x log x)$.
