What is Victor's greatest fear in not creating the monster? Does it stop him from delaying his trip to making the monster?

Victor is afraid his family will be harmed if he does not create the second monster. He does, however, delay.

I had now neglected my promise for some time, and I feared the effects of the daemon's disappointment. He might remain in Switzerland and wreak his vengeance on my relatives. This idea pursued me and tormented me at every moment from which I might otherwise have snatched repose and peace.