Concerning the fact that geeks tend to run counter to "classy" menswear, I'd like to oppose the living example of Vint Cerf, one of the forefathers of our current little technological cocoon. Guy works day-in and day-out with super-casual "shirt, jeans and relaxed suit jacket" types, and he insists on rocking the three-piece suit. If anything, Cerf has the kind of class that would allow him to ace the fedora.
Otherwise, I completely agree. Hats are being generally abused by a number of subcultures. The fedora tends to get a pass because it's now become a bit of a "Collegiate Artsy Type" accessory, or the type of clothing item Mad Men fans tend to flock to because Don Draper rocks one. Also consider Walter White's Porkpie. I don't know about you guys, but the year Heisenberg's Porkpie hat entered the public consciousness, my campus was *flooded* with would-be Cranstons that were twenty years too young to pull off the look.
It could be worse, though. The fedora still isn't the top hat, which has been shrunken back down to its feminine "pill hat" concept and generally Goth-ified to Hell and back, when it isn't worn by guys who deluded themselves into thinking they looked like Slash.
Personally, I stick to a toque in the winter and a soft bill cap in the summer. I'd honestly go bare, but having severe genetic predispositions towards early complete baldness makes it so I have to cover myself at nearly all times. If I don't have something on, on top, even inside, I continuously feel like my surroundings' heating bill needs to be paid ASAP. All my body heat escapes from there.