Answer
$a = 3, b = 5$
Work Step by Step
We have to find a solution of the following system of inequalities:
$$\begin{cases}
a^2-b^2<(a-b)^2\\
(a-b)^2<(a+b)^2.
\end{cases}$$ Let's rewrite the system using the difference of squares pattern:
$$\begin{cases}
(a-b)(a+b)&<(a-b)^2\\
(a-b)^2&<(a+b)^2.
\end{cases}$$ Move terms on one side in each inequality:
$$\begin{cases}
(a-b)(a+b)-(a-b)^2&<0\\
(a-b)^2-(a+b)^2&<0.
\end{cases}$$ Factor $a-b)$ in the first inequality and use the difference of squares pattern in the second:
$$\begin{cases}
(a-b)(a+b-a+b)&<0\\
(a-b-a-b)(a-b+a+b)&<0.
\end{cases}$$ Simplify:
$$\begin{cases}
2b(a-b)&<0\\
-4ab&<0.
\end{cases}$$ Simplify more:
$$\begin{cases}
b(a-b)&<0\\
ab&>0.
\end{cases}$$ We conclude the following: $$(a>0,b>0,ab).$$ For example: $$a=3,b=5.$$
