Answer
See explanations.
Work Step by Step
a. Step 1. Given the function $f(x)=\begin{cases} x,\ rational-x \\ 0,\ irrational-x \end{cases}$,
we need to show that $\lim_{x\to0}f(x)=0$.
Step 2. For any given small value $\epsilon\gt0$, we need to find a corresponding $\delta\gt0$ such that for all $x$ in the interval of $|x|\lt\delta$, we have $|f(x)|\lt\epsilon$.
Step 3. If $x$ is irrational, $f(x)=0$ and we do not need to prove this case.
Step 4. If $x$ is rational, let $\delta=\epsilon$, we have $|f(x)|\lt\epsilon$, $|x|\lt\epsilon$, and $|x|\lt\delta$.
Step 5. Working the above step backward, for any given small value $\epsilon\gt0$, we can find a corresponding $\delta=\epsilon$ such that for all $x$ in the interval of $|x|\lt\delta$, we have $|f(x)|\lt\epsilon$ which proves that $\lim_{x\to0}f(x)=0$.
b. For $x\ne0$, since every nonempty open interval of real numbers contains both rational and irrational numbers, it does not matter what value $\delta\gt0$ is; we can always find an irrational number within $(x-\delta, x)$ or $(x,x+\delta)$ such that $f(x)=x\ne0$. While at the same time, we can always find a rational number in the interval such that $f(x)=0$. Thus the limit $\lim_{x\to c}f(x)\ne0$ where $c\ne0$. This means that the function will not be continuous at $x=c$ if $c\ne0$.
