For personal game of the year, not necessarily the game that I enjoyed the most, but the one that impressed me the most with what it did.
For a game of the year that I'd say deserves being game of the year, as opposed to just a game that impressed me, I'd compare graphics, gameplay, balance, audio, bugs/optimization, UI and every other feature of the game. Its the reason that last year I had a hard time picking a game of the year along these lines, though I settled for Deus Ex as whilst it didn't have the best graphics, its boss fights were crap, and it had little real replayability, it was well made, polished, reasonably balanced, had good overall gameplay, didn't look bad and all round felt like it deserved it, unlike other contenders like BF3 or Skyrim, both of which failed on different fronts.
This year... I might have an even harder time picking. I think I've bought 3 or 4 games this year, because they were the only ones that remotely seemed interesting, though out of them there are some that definitely drop off the list instantly.