I've always believed it to be part sci-fi. I explain why to my dad so I can get him to try playing Skyrim, but he's set in his ways. Won't play anything other than Halo.
But think about it. Read the in-world books!
In-world evolutionary theories regarding the origins of the mer and man races, planet and moon orbit cycles, archaeology, palaeontology and even more.
In a way, it's more SF than Star Wars. In the original movies they don't explain the force and don't talk about HALF of the workings of the solar systems and universes, languages and species as much as TES games do.
It's part of the exploration aspect of these games. In exploring the world, you encounter people that want to know more about it. People who are basically scientists. Some use real-world science, others use a tweaked version to explain away the magical side of things or to represent that some of these people may be wrong or some facts are lost over time. Science and fact is very malleable. Even in real-life.
I kept one of every book I could find in Oblivion and Skyrim. After reading a good few of them it was pretty obvious I was playing a fantasy that had a good amount of sci-fi elements to it.
If you can get a sci-fi horror and sci-fi comedy, you can sure as hell have a "sci-fi fantasy".