Not really the monster, but what happened. See, in Fallout 3 I fast traveled the the German Police Town HQ. I'd only walked past it and discovered it, so I had never gone inside. I spawned outside a door and immediately started walking around trying to find an entrance. I could have sworn I heard something walking behind me, but I just guessed it was Dogmeat or something following me. Well the footsteps got closer and closer until I turned around to confirm that it was Dogmeat when I saw this super mutant right behind me with his nailboard mid-swing and about a foot away from my head. Scared me half to death, but two shotgun blasts to the face and he was out. So I continued walking when I turned the corner and right as I was turning, I came face to face with a centaur and that also made me jump. I have the worst luck with stuff like that.
I also think that the scariest looking monster was the village chief in that burning barn from Resident Evil 4. He just creeped me out... OH and the Iron Maidens from RE4. When one of those came walking around a corner, I near shit my pants.