There will usually be stressful periods of time during college. We all face some sort of adversity and we all deal with it in our own way. I usually am one that will be stressed out during a big test, but colleges have help anytime you need it. Professors have office hours that you can visit and get extra help, and some colleges have a "Math Center" (or something similar) where tutors are readily available throughout the week at all times of the day to answer any questions you may have. I took advantage of tutors help during many of my math courses and it was a great help. If math is what interests you, by all means go for it! Don't let the idea of stress scare you away.