I really like tiny cute hats, I have one for a christmassy costume, but I doubt that would look very nice on a guy, it's more like a hairpiece than a hat. Most guys dont suit hats because the clothes they wear do not fit with the hat... You could always take inspiration from Adam Savage (MythBusters), cause that type of hat works better than most with a casual outfit... When I see a guy with a fedora it's usually accompanied by tight jeans (frequently in an absurd color) and multi-colored shirts. That automatically has me think "douche"...
If your kind of the fit military type a Boonie is always neat, and works in both rain and sun... But thats maybe a bit too much like a cap, more a practical piece of headwear rather than a decorative one. You COULD just do like me though and wear mostly cat ears instead, those are cute and fun to wear, and I bet you'd get even more positive attention if you wore some as a guy than one does as a girl.