Chapter 5 - Polynomials and Factoring - 5.2 Factoring Trinomials of the Type x2+bx+c - 5.2 Exercise Set - Page 317: 56

$(a-3b)(a+b)$

The 2 numbers whose product is $ac= 1(-3)=-3 $ and whose sum is $b= -2 $ are $\left\{ -3,1 \right\}.$ Using these two numbers to decompose the middle term of the given trinomial, $ a^2-2ab-3b^2 ,$ then the factored form is \begin{array}{l} a^2-3ab+1ab-3b^2 \\\\= (a^2-3ab)+(1ab-3b^2) \\\\= a(a-3b)+b(a-3b) \\\\= (a-3b)(a+b) .\end{array}