Doctor What said:
It's been my favorite since I first saw it at the age of five.
That's pretty much the same for me, and I think this is the key here. Older Star Wars geeks, like in their 40's-50's, who got to see the trilogy live, probably felt a little cheated getting the more kid friendly RotJ after the darker ESB. From a kids perspective though, it's reverse as Return of the Jedi is the best while Empire Strikes back was a little weaker (pre-prequels of course). So while I can see the difference now it's still hard for me to look at Return in a negative way.
...Going a bit off topic but kind of the same thing with Bond movies I guess. I went through a James Bond kick when I was around 12 or 13 I think, watching all of them in a summer. Among my top 3 favourites would be Moonraker, while On Her Majesty's Secret Service was the worst. So having not watched them in 15 years it's funny now to see people talk about them and the general consensus is the exact opposite of my childish preferences.