Yes, it is possible to follow your heart and be financially secure. My early experience was to hold down a secure job, as a social worker with civil service benefits for a decade, while developing the career I really wanted by freelancing as an editorial photographer on the side. It did take some juggling for a while to do magazine assignments that didn't conflict with my regular work hours, however, but generous vacation benefits permitted annual foreign trips to allow developing a specialty as a travel photographer. I expect it would be easier to do this as a freelance writer, which I have also done occasionally, as the hours to write would be more flexible.
As a writer you could incorporate your interest in history and travel while you research your writings. I have several travel writer friends and unfortunately payment for articles and publishing venues are in sharp decline. Nevertheless, there are a few prolific writers who do very well, especially those who think outside the box. I would suggest you study the writers who do full-length books such as Bill Bryson, Simon Winchester, Paul Theroux, and Pico Iyer. Also, alternatively, you might try the direction my sister-in-law, Lynda J. Cox, has taken as the writer of Western romance novels while teaching university classes in creative writing.
Many people will tell you to be realistic and settle for security. But without risk there is no reward. Aspire, aim high, and persevere.