Was about to say the same thing more or less.Alcohol is a social lubricant.
It eases tense or awkward situations.
If you're with friends, that shouldn't be a problem.
Also if you are like me you drink for taste as well and its more fun done in a social setting.
As for the op it seems you were seen as being unsocial, not for the fact you refused to drink but for the fact you refused to join in on the group activity.
I guess a bit like not joining a game of monopoly a load of your friends are having.