Body harvest, hands down, no question about it finest game ever made. Not to mention it was quite obviously a precursor to GTA3 and to a lesser extent things like the first Halo and considering those two are responsible for inspiring the large majority of the games made in the 2000's I'd say that it's pretty important in the history of gaming, even if no one ever played it.
Ocarina of time was obviously fantastic. As was Mario 64. Both are probably in my top 10 games of all time.
I'm surprised to see so many people saying Perfect Dark over Goldeneye, maybe it was just my friends but I was under the impression that the rabid jerking off of Goldeneye knows no bounds. Don't let that come off as pretentious or anything, I'd be in the front row waiting to take a load in the face.
Also, don't judge me but Star Wars: Episode One Racer holds a special place in my heart.
EDIT: one more thing, I remember, first time around having a bomberman game with quite a good single player mode where you had to kick bombs under holes and stuff to solve puzzles, in 3d and everything. I recently bought bomberman hero assuming since it was a bomberman with a single player that would be it but alas it was not. I suppose the question I would ask is, 'Were Bomberman 64 and Bomberman hero the only two titles in that series on the N64 or is there another game that it could be that I remember so fondly from my childhood.
Edit 2: Remember Space station Silicone Valley? That rocked pretty hard too.