The only people who can rock fedoras are bad guys in Indiana Jones movies, Hard Boiled private dicks, and the occasional comic book character. Everyone else please stop.
i think the main problem is that fedoras have become an icon of irredeemability -- that's kind of mysterious and exciting when tied to a person inextricably linked to the criminal underworld, supervillainry, or the odd nazi -- it's not so ambiguous on top of the head of a pudgy, unemployed person.
A problem is that a lot of nerdlingers and dorks and dweebs think a fedora is an instant cool-making fashion accessory, even if they're simply wearing it with their t-shirt ordered from ThinkGeek and their extra-baggy cargo jorts.