Answer
$25x^2y^4-10x^2y^3+x^2y^2$
Work Step by Step
The square of a binomial sum is: $(A+B)^2=A^2+2AB+B^2$.
The square of a binomial difference is: $(A-B)^2=A^2-2AB+B^2$.
Hence here: $(5xy^2-xy)^2=(5xy^2)^2-2\cdot5xy^2\cdot xy+(xy)^2=25x^2y^4-10x^2y^3+x^2y^2$