If you mean hardest to get into, then according to https://medschoolinsiders.com/medical-student/top-5-most-competitive-specialties-in-medicine/ it's the 3rd hardest to get matched in Med School.
I've worked with neurosurgeons in the past, and it is a tough, tough specialty. Neurosurgeons perform a variety of procedures on the brain and spinal cord that require mastery of vascular, soft tissue, and orthopedic techniques. It's not unusual for a single, relatively "routine" case to take 7+ hours, and you'd likely perform multiple surgeries in a day. Given the parts of the body neurosurgeons work with, there's typically very little room for error and a high degree of unpredictability. With that said, as a neurosurgeon you have the opportunity to forever change peoples' lives--remove pain, restore function, and stave off life-threatening conditions. I'd recommend that you check out some books by neurosurgeons, such as "When the Air Hits Your Brain" by Dr. Frank Vertosick to get first-hand accounts of what the profession entails. Hope that helps!