I was a professor for a couple of years, and I will say that talking in front of a group of people (and being expected to answer all their questions) can be scary. It helps that by the time you become a professor, you should be well-versed in your field. Of course, you may still experience impostor syndrome ("I don't really know as much as they think I do!"), but this fades with time.
If you have a fear of public speaking, I recommend being prepared. I know that I don't always think well on my feet, so having plenty of examples and information ready before lecturing really helped.