I've always loved fedoras. I just never wear them for the same reason I never wear any sort of jewelry; I just don't like having stuff hanging off of me. I don't even wear a watch.
When I get married one of these days, I plan on tattooing the wedding ring on my finger rather than wearing an actual one.