Chapter 11 - Rational Expressions and Functions - 11-1 Simplifying Rational Expressions - Practice and Problem-Solving Exercises - Page 668: 49

$=\frac{a-3b}{a+4b}$

$\frac{a^2-5ab+6b^2}{a^2+2ab-8b^2}$ $=\frac{a^2-2ab-3ab+6b^2}{a^2+4ab-2ab-8b^2}$ $=\frac{(a-2b)(a-3b)}{(a-2b)(a+4b)}$ $=\frac{a-3b}{a+4b}$ The denominator of the original expression is 0 when $a = 2b$ or $a = -4b$. So the simplified form is $=\frac{a-3b}{a+4b}$, where $a \ne -4b$.