As a guy, I couldn't care less actually. There's a point where it looks weird though. If you have full and thick chest hair or none at all, they both look OK. It's when you get chest hair like mine, which is thin and doesn't cover well, that it looks...weird. My friend had such a problem, but never shaves it or anything. It's not too much of a problem since I've only ever seen him without a shirt on wen we go to the pool or something.
I shave mine, mostly because I'm whipped and my gf doesn't like it, and partially because it doesn't cover well and looks weird.