I believe there are two fundamental ways to view your job.
- You can have a job doing what you love.
- You can have a job that gives you the resources to do what you love.
For example. I really enjoy my work. I work for a great company, I work a lot of hours, and when I am successful at work it makes me feel great. I have a pretty limited amount of personal time, but that is ok for me. On the other hand, I have a friend who doesn't like his job, but it gives him a lot of personal time and pays him the money he needs to pursue his hobbies and interests outside of work.
So you need to ask yourself some questions. Does work or personal time make you most happy? Which job is most in line with what you want out of life? It sounds like both would be job jobs, so which job is harder for you to walk away from?