I've given up with anime lately, mostly for these reasons.
1: The manga is often better. I've heard that this is due to the fact that since manga is more accepted than anime in Japan (as anime is seen as the equivalent of cartoons over here, something that you might watch a more mature version of sometimes or see a bit of, but following them after a certain age seems immature) so the animated adaption is aimed towards a younger audience. They also like to cut unnecessary sections as well and if the series isn't a sure fire hit, it will be aimed towards a shorter run. You also then have the problem of adapting anything to another medium as well, with a lot of little things being lost including the stuff you filled in yourself (such as the voices).
2. A lot of it is indeed crap. Popular stuff is popular because it's either safe or it just happens to appeal to a large audience. More niche focused material is almost always going to be better, if only due to the fact that it can focus on it's task rather than pleasing a larger base. However, this runs the risk of not hitting the proper niche during it's run (becoming a cult classic), underperforming (niche being too small), being stuck in a certain rut or risk dying (if you try to expand outwards after you've reached your limits with a certain area, you risk upsetting the original fans, and the original stuff might not appeal to the new fans), etc.
3. It requires a lot of time/money/effort to watch. This is true of any show, but anime in particular. If you decide to watch a single episode, you have to commit half an hour to that, maybe more if you're watching online. Speaking of watching online, assuming you're not watching via hulu or something (where you get the ads as well), you've got to learn to properly read subtitles for a lot of series, wait on loading screens, and find a proper site once you've committed (often you'll hit a lot of ad heavy sites or worse). If you decide to support the official release, well, since you're part of a niche it's gonna cost you a lot more than normal. And then you have to deal with whatever the localization changed as well, in addition to the original changes in the paper to animation transition.
4. I get bored with it. This is mostly the summed up version of all the other points. Often I'll read the manga, and not even really want to buy it to reread, so I don't feel the need to watch either the animated version of it, or the bastardized original story featuring a lot of the elements from there. When I do, it's often one of the much smaller series, which means it's gonna be a lot shorter than the safe shounen series that's running into its triple digits. And then I've got to dedicate half an hour per episode, which I could be using to replay some of my games (several of which are waiting right next to me, stacked in order of want to replay), or do work in the form of studying or writing stories, or starting to watch a movie I like. My days already seem short enough, even with a mild case of insomnia, so I doubt fitting in a series would be an easy task.
Come summer though, I sometimes have the time, so I'll watch a series or two, but that's about it.