Why was prayer considered by some in the black community to be counterproductive to the civil rights struggle?

Some people in the black community thought that action was just as important as prayer.... prayer alone hadn't solved the community's problems, maybe action would.