Comparing an N64 and a PS3 is a little bit like comparing a Porsche to a Vespa; they're both fun if slightly impractical. One's inexpensive but with an impending strike-down by the law of diminishing returns, the other's expensive up front, but may earn value over time.
The upshot of getting an N64 is that you can buy most of the best games for less than 20USD at a local game store. However, while the N64, as of now, has a larger library of games than the PS3, the fact is that most of the games for the N64 really weren't all that good, and most of the ones that were good haven't aged half as well as they should have. Frankly, I'd be hard-pressed to come up with a Top 5 list of recommended exclusive games: 1)Hybrid Heaven 2)Tie: the Two Zelda games 3)Shadows of the Empire 4)Paper Mario 5)...Um... There, you see? And I can't even really recommend Hybrid Heaven that highly. I have fond memories of my N64, but they're few and far between.
If the concept of reincarnation applies to game consoles, wherein the XBox 360 is the "Dreamcast 2", the PS3 is the "NeoGeoDVD" (or rather, "NeoGeoBrD"); it's huge, it's expensive, it's got a painfully small library of expensive games (most of which are already available on other consoles with little or no discernable graphical difference), and a pretentious marketing campaign. Then again, you probably wouldn't be UNhappy with a PS3; after all, if you're going to pump that kind of cash into a console, your brain will lapse into "justify all" mode and you'll find yourself singing the praises of even the most sub-mediocre title.
The XBox 360 has become the "all-rounder" of this generation, an honor once held by the PS2, so picking between an N64 (unless you mean a Wii) and a PS3 is going to be a very unique and almost esoteric console ownership experience.
Final Word: I had an N64 and I liked it. I've played around on a PS3 and was thoroughly underwhelmed. You can buy 3 N64 games for the price of 1 PS3 game. The N64 has brought to the world some of the greatest games of all time, the PS3 has brought us virtually nothing... so far. Bargain or Investment, you decide.