It depends on the game.
If it's an RTS, I start very hard/highest because I'm generally good at them. (like, DoW II I started on primarch or w/e it's called)
If it's an FPS, I start at very hard/highest because I'm generally good at them. Sometimes I gotta pull back a bit, like if I went back to play UT2K4 through again and put it on Godlike, I'm sure it would be a struggle, but I'd still get pretty far and the game is supposed to be challenging anyway, so yeah, very hard/highest.
Most top down shooters I start on very hard/highest, but something like Ikaruga, well, you need to build up to something like that, so you start out on easy because the game is only like 30 minutes long anyway, and slowly work your way through until you can clear the game with 1 credit on all difficulties.
RPGs are generally not too difficult, so I start on very hard/highest, especially in a JRPG because I was always better at tactics rather than twitch gameplay, though I'm pretty good at both - it depends. Something like Baldur's Gate would be something I have trouble with at highest difficulty...Something like Fallout 3 is something I don't.
Racing games, well, I love time trials the most, and I shoot for a silver and then a gold and then whatever's above gold, but something like F-Zero GX, you play on Hard (highest available), then you unlock Master, and play through that. But I started that one on easy because you need to beat the cups on every difficult anyway to unlock certain things.
I guess I tend to go towards the hardest possible, and if I unlock a 'harder' mode, I tend to do it.
Only exception is if it's a "hard within reason" vs "IMPOSSIBLE TO PLAY THE GAME THROUGH THE FIRST TIME WITH" mode.
Like, say, in Doom, I'd play on ultra-violence, but not on nightmare because monsters respawn every 20-30 seconds or so, which is just aggravating. Same with Duke Nukem 3D, I'd play on Come Get Some, but not the difficulty above it.
Anyway, yeah, I generally put it on the hardest.