I'll have to give the nod to "Nocturne" and some of the other Shin Megami Tensei games. They pretty much feature everything mentioned here like monsters that can one shot your party through dumb luck, but also some rather tight item management since your limited in how many restoration items you can carry a lot more than many other games, and SMT is infamous for not handing out SP restoration items very freely, which means that you wind up having to horde them until the endgame and if you haven't it can rapidly become an exercise in Masochism.
One thing I WILL say though is that there is a differance between the game being hard, and it being fairly arbitrary. The big thing that makes Nocturne "hard" is the way it handles leveling up, and the fact that you can only have so many abillities and the ones you replace are gone forever. Without know specifically what your going to be facing ahead of time (say with a strategy guide) that paticular mechanic isn't so much hard, as it is ridiculous.
For more recent games I think Lost Odyssey was the hardest of the current gen, but isn't difficult compared to some of the stuff you've seen on the PS-2 (like SMT).
-
I am however going to include one stand out game that breaks these conventions though. Probably the hardest "Japanese Role Playing Game" is actually a Western RPG using a Japanese cultural/world setting (which ironically Japan hasn't seemed to produce much of). It was called "Deathlord" for the Apple II, you can find it on Abandonware sites but would need to run an emulator.
Deathlord is pretty much in the vein of oldschool ultima, with very simple graphics. Unlike later Ultimas (IV+) instead of recruiting companions you wind up having to create a party of heroes from a variety of classes, all of which have Japanese names. We're not just talking Samurai and Ninja here, it gets more obtuse in some ways than even the old AD&D Oriental Adventures book... and let's just say this game delights in torturing you.
Deathlord might be the hardest RPG of all time that isn't hard because of bad game design or shoddy programming or whatever.