men,s stiff collars wilted by nine in the morning.

It's the men's shirts. They're cleaned, heavily starched, and pressed. The shirt collars are stiff from the process, and the men look well groomed. Heat, sweating, ect., will make the collars wilt and not stand up straight. Thus they're considered wilted and the men are seen to be rather unkempt.