What Ben realizes that's has to stop the experiment, he thinks: "...it (the wave) had been an overwhelming success - too much of a success" what does Ben mean by these words?

Ben means that the Wave succeeded in emulating the way the Nazis were able to gain control and manipulate the populace. The Wave also succeeded in making Ben feel what that power was like..... as he was the leader.