Everyone gets all hyped up as soon as someone mentions Anonymous, and it's really not a big deal. Barring something obnoxious (stealing something they believe they have claim to, which encompasses a slogan, a logo, and possibly the word desu), something supremely arrogant that can be considered evil, or something stupid, the collective of "Anonymous" won't spring to action.
They aren't like a bunch of random mobsters, picking targets entirely at random to harass (most of the time, anyway). If you look at the bulk of Anonymous' attacks over the years, you can almost see their justification for what they do. They trolled Scientology because they believed it was standing in the way of free speech, they trolled Youtube because it was becoming too restricted and commercialized. There's even a new movement going on now to attack KnowYourMeme.com, because members of the collective believe that the site is drawing an unwarranted profit off their collective intellectual property (they've stated that it's not about Anonymous trying to collect money, but rather that it's not right for big business to be able to cash in on memes).
Also, I'd like to take the time to point out that Anonymous is not 4chan. Anonymous also isn't just the /b/ board on 4chan. Anonymous is a massive collective of people, spanning dozens of sites including ebaumsworld and an innumerable number of *chans (7chan, 711chan, 420chan, etc.). It's convenient to link this activity solely to /b/, simply because it's the most notorious *chan site, but the fact is that its own notoriety has driven most of the anonymous activists away from 4chan completely. In fact, I'd wager it'd be more accurate to link Anonymous with either 7chan or 711chan at this point, simply by the population of people there who do more than lurk, spam, or swap pornography.
In the end, I guess my point is twofold.
1) Don't get so worked up over Anonymous. Normal people rarely, if ever, cross paths with the people that comprise the collective, and when they do it's usually in the form of said normal person reacting to one of their attacks. The ones who actually need fear Anonymous' ire are those that speak openly against the "organization", or those that exude excessive arrogance or stupidity. (This is, of course, only a general rule. There are always incidents, like the epillepsy foundation attack, that break the paradigm.)
2) Don't associate Anonymous with 4chan, at least not exclusively. Anonymous exists on 4chan, yes, but only typically in the /b/ and /i/ boards. But you can find Anonymous in multiple other places, like ebaumsworld, Encyclopedia Dramatica, and the numerous other *chans. Anonymous is bigger than 4chan, and even when it was concentrated there, it was primarily only two boards.