I enjoy challenge; I don't enjoy frustration. Banging my head against the same segment for four straight hours is not fun. I experienced this recently with Catherine; it was a little disheartening, but for the sake of my blood pressure, I restarted on Easy.
On my first playthrough of a game, I'll usually start on Normal. If it's too hard, I'll kick it down a bit. If I beat it without much difficulty, and I feel like playing again, I'll take it up a notch.
As much as I enjoy fighting games, though, I'm just not very good at them. I mean, even on Very Easy, I have difficulty beating Street Fighter IV. But, then again, fighting games pretty much always have killer final bosses; on Very Easy, SF4 doesn't pose much challenge until that point. But on Easy, I start getting my ass kicked around the 5th match.
But in games that are a bit less frenetic, or at least have more of a flow, I can take it up to the harder difficulties with a bit of practice. Bioshock allows for careful planning, which lets me think through situations rather than just react. Racing games, once you know the course, it's just a matter of small adjustments. And music/rhythm games, I just get into the zone, and the performance flows out of me.