Tekken 5 and Soul Calibur 3 are both great fighters that are most definitly worth playing.
Rouge Galaxy is made by the same guys who did Dark Cloud, and I think they had a hand with Dragon Quest 8, so if you like one of those games, you'll most likely like them all.
Onimusha has about 4 games on the PS2, and they're all good, if a little too similar by number 4.
Any Ratchet and Clank game not called Deadlocked is so worth it.
Sly 2, not the first, not number 3, Sly 2, is one of the best games on the system, and one of my personal favourites.
*insert mention of Jak and Daxtor here*
Zone of the Enders, if you can find it, is great, but it's squeal, The 2nd Runner, is better in every way.
Maximo, Shinobi, and Rygar are all old, niche titles, and some of the best games to fall under the radar in ages.
Oh, and Black, easily the prettiest game on the system, makes Halo 3 look outdated, and it's not a bad game, either.
Of course, Grand Theft Auto has a good reputation for a reason, and San Andreas is a good game by any standards, but the Xbox and PC versions are much better.
SSX 3, Beyond Good and Evil, and Psychonauts also fall under the "get it if you can't find a better version" catagory.
And that's just the list of games that came up in a quick Google search, I haven't even touched the RPGs yet (get Wild Arms 5, it's a good one), nor have I touched upon the huge number of classics in the PS1 libary that are just that much better on a PS2.