Why does Momma get so upset with Marguerite's use of "by the way"?

Maya returns to the store, very happy; she brings tea cookies to Bailey, and tells him so. Maya uses the phrase "by the way," which gets her and Bailey beaten; her grandmother thinks that "by the way" is a phrase taking the Lord's name in vain, which is obscure and is hard to understand for the children.