Depends. I really wanted to send back Duke Nukem Forever (I only rented it thankfully) as I'd had enough with it, but because the achievements were generally easy to get or I knew I could get them, I continued with it. Achievements are usually my motivation. Having said that, there were a few achievements in DNF that were unfair and very frustrating. I shouldn't really have let the game get to me so much when I clearly didn't like it. But I just can't bear to leave a game unfinished achievement-wise.
However, I did do that with Final Fantasy XIII. I should never have bought that game when I knew I don't even like JRPGs. God knows what got me to buy it... If I enjoyed the gameplay, characters and story (the story I was sort of alright with, but the gameplay felt like a chore) some more, I probably would have finished it.