Answer
$xy(x+y)(x-3y)$
Work Step by Step
Factoring the $GCF=
xy
,$ the given expression, $
x^3y-2x^2y^2-3xy^3
,$ is equivalent to
\begin{array}{l}\require{cancel}
xy(x^2-2xy-3y^2)
.\end{array}
The trinomial expression above has $c=
-3
$ and $b=
-2
.$
The possible factors of $c$ are $
\{ 1,-3 \}
,\{ -1,3 \}
$. Among these factors, the pair whose sum is equal to $b$ is $\{
1,-3
\}.$ Hence, the factored form of the given expression is
\begin{array}{l}\require{cancel}
xy(x+y)(x-3y)
.\end{array}