Yep, Grave of the Fireflies. Still hit just as hard the third time I watched it.
Hotel Rwanda was pretty sad too, but the big thing about it was that it made me seriously question whether I would participate in the same kind of atrocities as the Rwandan genocide if the only alternative was being killed myself.
The first 10 minutes of Up was pure concentrated sadness, but luckily most of the rest of the movie was funny and upbeat.