Not to discourage you at all. Writing here only because you asked :)
Downsides are like in many other industries:
- work long hours
- be available when client wants
. Night and weekend deadlines
- agreement with peers and managers even if you are better expert than them
- lesser opportunities and industry growth. See Bureau of Labor Statistics reports.
- setup in a company could be a one boss show. Everyone agreeing and doing what a single person instructs.
- expensive investment if you want to be self employed, considering equipment and gaining clients as well.
- patience with iterative work. Ideas and wants keep changing till there is perfection.
The biggest downsides are usually long hours and tight timelines because there aren't a lot of animators on a project, especially in a corporate setting or obviously if it's just you freelancing. The best way to combat this is to be open and honest. For example, "I can get this to you by next Wednesday, but here's what will be missing." Be very upfront about timelines, "I need 15 business days to complete this" etc. And ALWAYS give yourself a buffer. If you think it'll take you 3 days, say 5. It's always better to provide the end result early, but have a backup in case something happens (program crashes - yes this happens). You'll need to be very self-motivated to provide sooner than deadline.