I don't think it's about being different, but about being misunderstood or not understood at all. And obviously, that IS a kind of difference, but it's important what kind of a difference it is:
First, it has to be "over the limit". So we are all different, but we can accept some amount of difference, especially when we expect it. Someone likes football while I prefer basketball? Who would even think of the other guy being "different" and "creepy". On the other hand, if he says that he likes to kill and skin little kittens, and then eat the meat and sell the skin, then I freak out and keep my distance. Because that is outside of my scope of understanding, (so I can't understand the why of this strange hobby) and it is not included in my "set of things that people can like or do, even if i don't get it".
The other part is visible difference, that is "calculated" from the size of the difference, the importance of the part which is different and last but not least, how unpleasing it is to look at the difference.
So what do we think of when we don't understand something: "If it's weaker then me, then I should dispose of it, or show myself that i am superior, or if it's stronger than me, than gather around with like-minden people and talk about how wierd and whatever it is, while staying as far from it as possible." "It" can be a person or something else.
On the other hand, humans have an extremely powerful mind and can process stuff like difference when they actually want to. An open mind is a tad more dangerous usually than a closed one, but at least you won't automatically beleive everyone "strange" to be someone to stay away from.
And while I kinda liked magneto, I'd still probably go for the other side. I was bullied plenty and sometimes became a bully myself during school. I'm still kinda "suffering" from being different than your avarage youth, but in the end someone has to be the more intelligent, wise and whatever, and it would kinda suck if I, the super evolved guy, turned out to be inferior in terms of intelligence and wisdom. For we all know what power without wisdom equals to...