Assuming they are indeed within a reasonable range of you in terms of physical ability, I would say you shouldn't retaliate simply because responding to aggression usually just promotes more aggression, if you start fighting back and that guy doesn't back down, that fights gonna keep going until one of you ends up in a hospital.
To add to that, self defense is only really needed in small doses in a civilized world, typically if someone assaults you, it wouldn't be all that hard to get away from them, or get more professional help if it came to that.
Im not technically a die hard pacifist, I can think of situations where fighting back would be both necessary and practical, but I look at it the same way as I do firearm possession, if you go into that situation expecting and preparing to fight back, you typically only stand to make things worse.
Its not really an issue of right or wrong, more of a matter of making things worse. On a final note, if I was actually trained in the proper way of subduing or incapacitating an aggressor without causing severe bodily harm, I would probly do that if need be, those techniques exist for that very reason and serve people well, but in my current state my only real options are back the hell out and avoid a scene, or start flailing my dumbass around in an attempt to hit a guy, which I don't foresee going well.
TL

R, if you go into a situation with the intent and preparation to hurt someone, people are gonna walk out of that hurt, and violence is a chain reaction.
Edit: very, VERY loosely related, but mostly for lulz.
http://www.youtube.com/watch?v=ipYkuCZ2IYI