The upper crust refers to high society, social or financial elite. The term wasn’t used to refer to the aristocracy until the 19th century. Bread was made into flat loaves and was baked in stone ovens. These loaves would be sliced horizontally. The lower crust was inedible because it had been in contact with the oven floor and was to hard and dirty to eat. It was sometimes used as a plate or given to the servants to eat. The upper crust was baked just right and was clean. This was given to the lord and lady of the castle or manor to eat. These people became known as the upper crusts.