I am so sorry that the adults in your school have let you down. And I say that because they should have a steady "pipeline" from middle school feeding into the high school band program. Whoever is supposed to create that interest dropped the ball. At first I was thinking of challenging you to fix it, but then I remembered, you are very driven academically. Trying to save band would take you away from your academics.
No, it's not bad if you quit band. And, you would not be quitting "it." It is quitting you. As you described it, it is falling apart. I doubt that any scholarship your teacher could help you get would be enough to even pay for your books. If anyone asks, you could just say that the band was starting to dis-band! (bad joke, sorry!)
What year are you in? I am happy to help you whenever needed. You seem to be under a lot of pressure to make decisions and commitments - happy to help. Life sometimes gets a little hectic.