how are the people from the fringes a threat

They have a large population and a lack of food, which causes them to raid civilized areas in order to survive. Parents tell children their children scarey stories about the fringe people like we tell stories about goblins and witches.