how did the casta system work?

Do you mean Caste System? Caste systems compartimentalise and commoditized, through slavery, different groups of people. Colonial invaders often ranked races and racial mixtures. Whiteness was the most sought after trait in these caste systems.