Chapter 3 - Exponents, Polynomials and Functions - 3.2 Combining Functions - 3.2 Exercises - Page 244: 37

$25x^2-80x+64$

The student just squared the first and last terms, which is not how squaring of binomials is done. Using the square of a binomial, which is given by $(a+b)^2=a^2+2ab+b^2$ or by $(a-b)^2=a^2-2ab+b^2,$ the expression, $(5x-8)^2,$ above is equivalent to \begin{array}{l}\require{cancel} (5x-8)^2 \\\\= (5x)^2-2(5x)(8)+(8)^2 \\\\= 25x^2-80x+64 .\end{array}