In my experience, it's usually somewhere around atmosphere and immersion. This can be from may different sources, including visual design, story, music, etc. It's just something about the feel of a game that really does it for me. Some of my favourite games:
Phoenix Wright series, Ape Escape, Halo: CE, Grandia, SW:KoToR, Mercenaries, Tombi 2, Borderlands, Metroid Prime. These games all managed to really pull me in, and give me some kind of emotion either during gameplay or afterwards, but usually both. It's possible for me to thoroughly appreciate a game's design but not think a huge amount of the game (Portal, Civilization, Total War series.)
Of course if the controls or graphics are distractingly bad, I will lose touch with the game, and a game doesn't need a great atmosphere for me to love it (Puzzle Quest) but something about those games I listed up there just really makes them stick in my memory with positive emotions. I may have forgotten some.