When the scripting is as loose as possible so that there are as few predictable moments as possible; Stalker is a great pioneer of this with the A-Life system, which means besides a few story-important bits, most of the creatures can appear anywhere, often to your severe detriment or perhaps to your benefit (I had a controller wander into a firefight once and murderise half the enemy). In Pripyat, I now dread going out later in the game because a chimera could be anywhere; I've met them pretty much everywhere at least once, on one occasion massacring my NPC allies during a mission...
Soundtrack and scenery are pretty good too; if they make me tense, then all the better, again Stalker does a good job of this. Pripyat is one of the scariest locations I've ever seen in a game; much of it is down to the overgrown nature of the city and the really eerie soundtrack.
The enemies contribute too; again, Stalker (you may notice I really like this series) has some enemies who you feel are significant threats, either because they are psychic and can attack you in a different room, or because they can go invisible or because they are built like a tank and require several hundred rounds of ammo to take down.
TL;DR - a bit of everything helps really; the atmosphere is probably the element I find most unnerving; make it eerie and you can put pretty much anything in there and I'll jump.
