I think you're asking an important question and the answer will always be different. What may be difficult for one person will be easy for another.
My advice, remember that you have the freedom to decide what direction you would like your career to go. Try things out, continue to ask questions and find out what works best for you.
For me, work/life balance WAS the most difficult aspect of my job. At times, adapting to the culture was quite difficult, because I have a teaching and business back ground and dominant personality.
However, I am quite satisfied with what the field has to offer and there really isn't anything challenging to me anymore. I've overcome these obstacles and I am grateful for the opportunity to govern my career the way that I see fit.
Here are some of my opinions based on my experience in the hospital.
The medical field itself has a universal culture.
For example, medical terminology and medical asepsis, surgical technique...etc. This language and practice is shared at every hospital, surgical center and healthcare facilities across the US.
It is imperative that as a healthcare professional you are familiar with the culture, hospital rules, and hospital politics in order to effectively work with other healthcare professionals.
This can be the most difficult aspect of the job, as you'll have to adapt. Aside of universally shared terminology and other universally shared items, the culture, politics and hospital rules will be different from hospital to hospital and department to department.
Getting along isn't always the easiest thing to do and I think this is true in any career.
Work Life Balance:
The medical field is staffed 24/7, so once employed, you can work endless hours and have multiple jobs from freelancing, per-diem, registry travel contracts, etc.
At a time, I worked inconsistent shifts, lots of over-time which was sometimes forced overtime when I was the only tech available.
That period of my life was a "BURN OUT", but I earned six figures. Bittersweet.
I would say that if you allow your work-life balance to become UNBALANCED, then this will be the MOST DIFFICULT aspect of the job.
I don't consider skill development in radiology to be difficult 'after' the learning curve.
For example, learning to operate multi vendor equipment for different purposes, from fluoroscopy to x-ray to the operating room takes time and how quickly you master it will vary from individual to individual.
However, I think 'anyone' who is dedicated can master these skills and become a valued staff member and contribute enormously to the community.
I hope I thoroughly answered your question.