Calculus: Early Transcendentals (2nd Edition)

The solution is $$f(x)=x^2-6.$$
Since $f(f(x))$ has to give a polynomial of 4th degree we will seek $f(x)$ in the form of a polynomial of the 2nd degree, i.e. we will demand that $f(x)=ax^2+bx+c$. Let's determine $a,$ $b$ and $c$: $$f(f(x))=f(ax^2+bx+c)=a(ax^2+bx+c)^2+b(ax^2+bx+c)+c=a(a^2x^4+b^2x^2+c^2+2abx^3+2bcx+2acx^2)+abx^2+b^2x+bc+c=a^3x^4+2a^2bx^3+(ab^2+2a^2c+ab)x^2+(2abc+b^2)x+ac^2+bc+c=x^4-12x^2+30.$$ Equating the coefficients multiplying same powers of $x$ we get $$a^3=1\Rightarrow a=1;$$ $$2a^2b=0\Rightarrow2b=0\Rightarrow b=0;$$ $$ab^2+2a^2c+ab=-12\Rightarrow 2c=-12\Rightarrow c=-6.$$ The rest of the equations can be used for checking. Indeed, we have $$2abc+b^2=2\cdot1\cdot0+0^2=0;$$ $$ac^2+bc+c=1\cdot3^2+0-6=36-6=30.$$ Finally $$f(x)=x^2-6.$$