In the case of something like a horror game, what you hear but can't see is scarier because you might not know what is making the sound, where it is and where it is coming from. It's that fear of the unknown, making it worse in your mind. Imagine seeing it first... you'd know what it was and what is happening about a second after you've already pulled the trigger. Doom 3, for instance.
In the example of Sleep Paralysis, what you see would be scarier because sounds are away from you, while visual hallucinations are IN THE ROOM WITH YOU. Your brain automatically trusts your eyes more than any other organ, so trying to convince yourself that what you're seeing isn't real is a lot harder than convincing yourself that what you're hearing isn't real.
I do not envy you.