Mario and Yoshi was the first Game Boy game I ever got and it's still around here somewhere...
Destruction Derby for PS I think was the first video game I ever got which wasn't Solitaire or MineSweeper, still got it somewhere around here too. We had a game called Klonoa: Door to Phantomile which we lent to a family friend and never got back. It was the best game I'd ever played and so when we discovered Ebay I was so excited to get it back. I also opted to get all the original Megaman games for Game Boy and I've so far managed to get the first two. I've also got Crash Bandicoot 1, 2 and 3 as well as Spyro 1 and 3 for PS. I've got some old Game Boy games which have old Atari games on them like Asteroids as well.
I really don't like getting rid of my old games because I KNOW I'm going to want to play them again. Every now and then I'll get bored of the game I'm currently playing and, because I can't afford any new ones, I'll go back through my old ones and see if there's any I'd like to re-play. I still play Crash Bandicoot every now and then and I restarted my Pokemon Red game (which I've found to be more monotonous than I remember, but still good). I've kept every Game Boy I've ever had (which is pretty much every one that's come out up till the DS) and I still play my PS2 frequently and found no need (and no money) to upgrade, although now they're bringing out Ratchet and Clank games on the PS3.
However, if you want REALLY old, I've got these machines called Vetrexes which are basically TV's with a controller that you plug games in the side of. They have Atari style games and I still like to take it down out of the cupboard every once in a while and play.
Ah...nostalgia...