I side with the Imperials. Not because I agree with everything they've done; heck, I've even considered siding with the Stormcloaks because I think the Empire deserves to collapse. That said, the Stormcloaks are NOT the right people to take Skyrim back. I firmly believe that one should "never give power to angry people," and the Stormcloaks are nothing BUT angry.
People often counter the whole "Wildhelm is a slum" point by saying "well, that's just Ulfric and he's focusing on the war." Have you looked at his supporters, though? Laila Law-Bringer could not be any more of a misnomer; she has no power in her wretched hive of a town and she's clueless to this fact. Korir, Jarl of Winterhold (as in the College of Winterhold), hates magic to the point that other Jarls ignore him because of his anti-magic ranting. Then there's Skald the Elder, a tactical Leeroy Jenkins, who has put so many troops to the Stormcloak cause that his city is practically undefended. Yyyyeah... the Stormcloaks have pretty cruddy leadership as a whole.
At least the Imperial Jarls are fairly likeable, or at least Balgruuf, Elisef, and Brunwolf (replacement Jarl of Windhelm) are. Heck, Elisef even worships Talos in secret. These people are worthy of ruling their cities, and by supporting the Imperials, I support them.
Really, what I dislike most about the Empire is the Emperor who accepted the White-Gold Concordant... and, heh, well...