Aye, to be sci-fi means, generally, either no magic or very minor 'magic', usually by another name. Often, "The Power of The Mind". Either way, it probably won't be in the foreground. Example: Dragonriders of Pern. The dragons are (telepathic) genetically engineered beings, and society is surviving on another world after their spaceship from earth landed there. (Additionally, the stories are much more focused on the systems of society and the lives of its inhabitants than epic adventures and good vs. evil)
The Elder scrolls, however, have magic very much in the foreground, are centered around epic adventures, and basically have all the elements of fantasy, not sci-fi.
...
They key to this is that there is having a Sci-Fi setting, and a Sci-Fi tone, and these are two different things. Are there spaceships, advanced technology, etc.? Then the setting is Sci-fi, and if instead the strange or fantastical elements of the setting function on a 'magic' then the setting is more fantasy. If the setting is then assessed, its implications and intricacies worked out and explored, often with a heavy focus on the human condition and not good vs. evil, the Tone is then more Sci-fi. If it appears at first to be good vs. evil, but the 'evil' is then found to just has a different system of morals, or a problematic communication barrier (the Ender series, for example) this also works.
There are works aplenty, however, of Sci-fi setting stories with more of the themes or tone of fantasy; Star Wars is a common example (after all, the Hero's Journey that Lucas used is a Fantasy staple since time immemorial), and the somewhat rarer bird of a Fantasy-set series with more of the tones traditionally associated with Sci-fi (I would argue that A Song of Ice and Fire and The Kingkiller Chronicles are a couple examples of this); This murky middle ground is what leads to the conception that sci-fi and fantasy are the same creature dressed up a little differently, but I disagree. As you leave this area, headed one way or the other, they become very different ways of discussing the not quite real.
(in my humble opinion anyway)