Answer
$x^2\dfrac{\partial ^2 f }{\partial x^2} + 2xy \dfrac{\partial ^2 f }{\partial x \partial y} +y^2 \dfrac{\partial ^2 f }{\partial y^2} =n(n-1) f(x,y)$
Work Step by Step
We have: $\dfrac{d(tx)}{dt}=x$ and $\dfrac{d(ty)}{dt}=y$
$\implies \dfrac{d^2(tx)}{dt^2}=x; \\ \dfrac{d^2(ty)}{dt^2}=0$
Now, $f_x[\dfrac{d(tx)}{dt}]+f_y[\dfrac{d(ty)}{dt}]=nt^{n-1}f(x,y)$
$[x(xf_{xx}+yf_{xy}+(0)f_x]+[y(y(f_{yy}+xf_{xy})+(0)f_y]=n(n-1) t^{n-1} f(x,y)$
$\implies x^2f_{xx} +xyf_{xy} +xy f_{xy}=n(n-1) t^{n-2} f(x,y)$
Set $t=1$
So, $x^2f_{xx} + 2xyf_{xy} +y^2 f_{yy}=n(n-1) f(x,y)$
Therefore, $x^2\dfrac{\partial ^2 f }{\partial x^2} + 2xy \dfrac{\partial ^2 f }{\partial x \partial y} +y^2 \dfrac{\partial ^2 f }{\partial y^2} =n(n-1) f(x,y)$
