I'll speak from experience on this one.
Back in high school, one day in gym class, a girl walked up to me and asked if I was gay. I said no. I then spend the rest of the entire day asking all my friends if I came off as gay or something.
Looking back now, I have no idea what I was thinking. I'm not homophobic in any way, I have gay friends, gay relatives, and I'm very pro-gay marriage. Like others have said, it's just a social stigma. Saying someone or something is "gay" almost always has negative connotations, regardless of one's feelings on the issue of homosexuality.
It's just something that we, as a society, have to get over. Not just men, but women too. I really don't feel like this has something to do with "manlyness", per se.
Personally, since high school, I've become much more comfortable with my sexuality. If someone were to ask me if I was gay now, I just say, "No, but sometimes I wish I was. It'd make things ALOT easier!".