Personally, i thought that teh Movie hit on this element pretty well
1. A mining company would not have access to the best weapons, thus the helicopters and a big flying thing that looked like a hovercraft from the Matrix (same with the mechs) in the place of Gundams, Jets and satelite cannons and why they had to rig a shuttle in the place of a real bomber. And wven their "dasiy cutters" were just flats of Mining explosives, rather than real bombs.
2. Just looking at the iraq war, the world has a problem when companies go into an area and mow down anyone who looks at them funny. Public opnion still matters and even teh RDA has to have some fear of thier stockholders who would take really badly to the slaughter of the blue cat peoples for no reason other than a shiny rock who's only real application is military. Besides, the NA'vi looked distrubingly human and were so pure in thier intentions and culture, that many would have a hard time justifying thier extinction.
3. THe Aliens have a signifigant little advantage, IE get a couple of thousand of them together and they can stomp over you "help is six year away" ass. Do the words, General Custer, mean anything. That and the Na'vi are ten feet tall and wield bows that have roughly the same amount of power as a ballista. they also have dragons, so there.
4. Teh scientists at teh base obviously had some fondness for the planet, so going on a rampage would have caused a little bit of division, something that is not good when you are 6 years away from the nearest supply depot.
5. It is much easier to try and make friends than fight a war. They tried that at first, with the schools and the english programs (things probably went bad about the time they broke out shakespear, or maybe that was just my highschool). If you think about what the RDA stood to lose if they went bannanas, it is perfectly reasonable that they tried the peace route first. that, and the peace route worked, until the RDA started killing the planet and eagerly eyeing hometree and rubbing thier palms together, cackling evily.
6. The RDA is a company, not an army, if you remeber it was Quartich, an EX military type, who basically hijacked the base when he was launching his attack.