I'm fine with them as long as the game is first person. In third person perspective it just feels awkward, but I can stomach it.
Now what really gets under my skin is when the writers think it's perfectly natural to have your silent protagonist engage in conversations where the NPCs ask you stuff and the game progresses as if you picked the answer they felt appropriate. I mean, it's fine to make the player project him/herself in their in-game persona, but if you're going to build your game around that, write the script accordingly.
The main offender? Shining Force. I love the series, but all you can say during the game is "yes" or "no", yet the rest of the cast acts like you just recited Shakespeare. In Shining Force 3 this gets even more ridiculous, as you play through the events of the game controlling various characters over the same time period, and only the guy you're playing as gives everyone the silent treatment. So when you change characters, your can see your old guy's dialogue. So it's not even for immersion's sake, it's like watching a movie and this one guy gets all his lines bleeped out.
(I'm not considering the MCs from games such as the Fallout series where you have set dialogue choices as "silent", mind you).