That's a good point. If morality were truly, completely relative, it would effectively render it meaningless. On the other hand, people who brandish the morality card as a simple, absolute truth are also wrong. On the most basic societal level, what is and is not considered moral really comes down to whether it improves quality of life for people in the society. Actions like murder generally detract from quality of life (people killed are now dead, and people still alive are now left in a state of fear), and are thus deplored by society. However, many societies still practice capital punishment (which if you define murder as "humans depriving other humans of life" is technically murder), but do so under the banner of improving quality of life for the other people in the society.
From a game theoretic perspective, cooperation and trade occur when players (people) stand to each gain more (or at the very least not lose anything) by working together than they would by going it alone. That's why job specialization is possible; otherwise we'd all be hunters or subsistence farmers. What many people call "morality" is really nothing more than baseless - and thus illogical - idealism (see: the battle over gay marriage). Real morality stems from the question: for any given action, is the result beneficial to at least some players without harming others? If so, said action is considered moral. If it neither helps nor hurts anyone at all, it is neutral. However, if it does harm some, however slightly, that's where the question of morality comes into play.
In Star Trek, the Vulcans' slogan is "the needs of the many outweigh the needs of the few." Logically this is sound; if many stand to gain from the sacrifice of a few, the combined benefit for society is more than the combined loss, and is thus a net gain. This is also true regardless of scale: "kill 1 to save 100" is equivalent to "kill 1,000 to save 100,000". NOW we're in the realm of relative morals. If all you care about is society as a whole, the aforementioned slogan makes perfect sense. If, however, you care about individual freedoms and to hell with "society", then it absolutely does not make sense. Although, if "society" collapses as a result of placing individuality first, everyone - including the individualists - are screwed. Also, if you care only about society as a whole, you could theoretically run into situations like "if I kill these 10 people in cold blood, the other 7 billion people in the world will get a single potato each." On a 1-to-1 scale, this is ridiculous; however, the COMBINED benefit of everyone who gets a potato could possibly add up to more than the loss to the 10 people killed, which would logically justify this admittedly stupid scenario. Therefore, the perspective that makes the most sense is somewhere in between. Morality has to be judged on a case by case basis, and it depends on who you care about. And there's nothing inherently moral or immoral in caring about one group more than another.