This is such a great question - getting into the world of professional sports is exciting to many, but very smart of you to consider the realities of what it would actually be like. I just wrapped up 7 years of working for a PGA TOUR host organization - meaning we host the golf tournament and the PGA TOUR provides the professionals.
Working in sports is a front row seat in the action and can be very rewarding. The one thing I would caution - is that it is more than a really cool job, as it is still a job. Sports tend to be a very demanding profession as many leagues gave games/tournaments on the weekends and holidays. Also, you are there to work - not meet the athletes and observe. If sports is your passion go for it - will be something that holds your interest.
It is best to identify a college with a sports management program, as they have the best long-term networking opportunities and so much of sports is about who you know.
If you're passionate about sports then the challenges won't seem like challenges but rather hurdles that you learn from and gain experience and perspective from. There are many different avenues to finding a career in sports. Are there certain areas you're passionate or most interested in pursuing? Depending on what avenue you take one thing you'll learn is that sports and your job within sports might require you to work on weekends, holidays, and/or longer hours than most. There are some sacrifices that you have to make when working in sports but if you're truly passionate about the career path then it's worth the investment!