I'm fairly comfortable with my body, so I don't mind taking off my shirt at the beach or pool, or whatever.
I reckon the whole "topless/nudity" deal should be a bit like casual friday. It doesn't happen often, but hey go for it when the time comes, like a "nudie day" or something that happens once every month or something.
As for walking around in public topless - here's the thing. I may in fact not WANT to look at someone's breasts, man or woman. I don't feel its a "right" of someone to say "Hey, I want to go topless, so fuck you and your prudish nature."
Guys wandering around without their shirt on in the street kinda creeps me out. I can't stop staring at their nipples (no homo). Its the same with women, and it just seems like they're drawing attention to themselves.
I don't think men or women have the right to walk around topless in casual public. I think it would be better if the laws were equal on a beach, or at a pool, because hey - less clothing is going to be off anyway.