Why does the mariner only tell his story to certain people?

The Mariner has to talk to whoever will listen, that's part of his penance for killing the albatross. People probably ran the other direction when they saw the creepy Mariner coming to tell his three hour story. The Mariner just happened to corner this wedding guest.

He holds him with his skinny hand,

`There was a ship,' quoth he.

`Hold off ! unhand me, grey-beard loon !'

Eftsoons his hand dropt he.

He holds him with his glittering eye--

The Wedding-Guest stood still....