The salary varies quite substantially. You are right that you will not find many private chefs making less than $50,000 per year; at the upper end it can be as high as several hundred thousand dollars per year.
Most private chefs have a career in a restaurant or go to culinary school beforehand to prove that they are capable of working for a family full time. So most private chefs have more experience than a chef beginning their career, to begin with.
So a real comparison is "experienced chef in a restaurant" vs "private chef (who has similar amount of experience)", and I think you would find that the salaries are not all that different. What is different is the lifestyle and potential perks.
The hours and headaches of serving many people and running a restaurant as a business, and the hustle and bustle, is what drives many chefs to work for as few people as possible.
But to do this you often need to "put in your time" at a restaurant before you can start looking to work for a family.
You can also be a corporate chef, which is a sort of in between of a restaurant and working in someone's home; the head chef of Google made several million dollars after Google went public, from stock options that were given out as a perk of cooking for early Google employees.