For me, it's usually when I'm done getting a cerntain number of achievements for a given game. For example, I "pawned" Left 4 Dead when I got the achievements I knew I could unlock without a dumptruck-load-of-dead-zombies amount of frustration. I got 935/1250 and for me that's a success.
Other games like Portal: Still Alive and Battlefield 1943 I've gotten all the achievements for them and I just stopped caring for them because there was nothing else to work towards.
-AND THEN-
Some other games like Turok, 170/1000, I stopped playing because it wasn't as fun as I thought it would be. The game just felt broken, the A.I. was very unforgiving, some of the dinosaurs were unbelieveablely cheap [[i.e. like the crocodile one who could climb trees and leap distances that would make the Hunters in Left 4 Dead jealous]], QTE up the WA-ZOO (most of which were triggered by the tree-climbing corcodiles), and a couple of the bosses were too difficult to beat and resulted in trail and error matches rather than an actual strategy.