Answer
A) $f(x)=x^2-3x$
B) $10$
Work Step by Step
A) The other two functions would not work with plugging in five. The second one would be undetermined, the third would result in a negative under the radical.
B) Plug in five and solve from there. 25 minus 15 is 10.