What is Ralph angry about? Why?

Ralph points out the other boys near the bathing pool and explains that Simon has worked as hard as he has at building shelters. The other boys, however, swim and play not fulfilling any of the responsibilities they had decided on at the meetings.