I can't speak to the prestige of any university apart from my alma mater, the University of California Santa Cruz, which was really good and has plenty of top-notch astrophysicists working there.
Wherever you go, be prepared to go all in for a PhD, and a career in academia and research, as this is where most of your options will be. If that is what you truly and strongly desire, and you have the courage and are prepared to put in the years of work required, then you will certainly succeed.
I spoke to a few UCSC professors about post-graduate career options. I was also an intern at NASA Ames research center for two and a half years where I got to ask a few questions to the research fellows who worked there. Astrophysics is a field with pretty much zero career options available to the bachelor's degree level of training, apart from teaching at a pre-college level. That is why I recommend going through a doctorate program and working to obtain a PhD as a few of my classmates did (one of whom was Konstantin Batygin!)
Note, if you go to grad school for a doctorate level degree, you will most likely have to work as a teacher's assistant, so teaching is something you might not be able to avoid (something to consider if that's not what you want to do).
Be that as it may, in the course of doing research work or side gigs at one's university of choice it's still possible to pick up a few skills and practices that one really enjoys, and discover a "plan B" career if your confidence runs out when finishing undergraduate work. I did exactly that and I can say that it is nothing to be ashamed of, but it won't necessarily involve much if any of the skills learned studying physics. The most annoying part about going this route is being asked by prospective employers why you "gave up" on your aspirations of becoming an astrophysicist, and why your degree is relevant or useful to their organization. This can really be a drag on one's self-esteem during a job interview, and requires prepared responses and examples of work experience outside of one's education.