Here's a better question: if you want to see The Avengers, why wouldn't you see the pre-films? If you don't like them, then you won't like the Avengers. But if you are the kind of guy who wants to watch a superhero team-up movie, then you have no excuse for not having already seen them.
In short, yes. By all that is holy, yes. It will help you understand it all better; though you'd probably kind of 'get it' without any background, it's going to be a lot better for those who already know what's going on. These previous films are the exposition for the main event in The Avengers. Also, they are just really cool movies.
Well, The Incredible Hulk was kind of mediocre, but the rest were great.