Aside from a literal interpretation, how else could this warning be constructed.

Literally, we have to concede the journey is dangerous......... but the warning comes from the fact that it's the most dangerous part of the journey....... Gandalf won't be with them.