It depends on the game
I usually start on medium, but there are games I only play on easy/hard. If I start on medium, I will continue playing until I have completed on each difficulty.
I started Ninja Gaiden 2 on the highest possible difficulty.(it's the only way this game should be played)
I refuse to play CoD4 on anything higher than medium.(the only change is that enemies become a bit more aggressive with grenades and your health is dropped to 1/3rd of what it is on normal)
I play Half Life 2 on easy since there's barely any difference between the difficulties.(enemies doing more damage doesn't matter when you know where the enemies come from and are able to kill them before they see you)
In any RTS, I only play skirmishes so I take the highest difficulty bot, and give it(if possible) "Steamroller" AI.