While better AI would help a lot of games, I don't think the problem is unintentionally bad AI. I think it's intentionally bad AI. Uncharted springs immediately to mind. The enemies on the highest difficult would literally stop and sometimes squat over grenades I'd thrown at them. I refuse to believe that was an accident. Instead, I think it was likely done so the grenades would feel powerful, even in the hands of unskilled players. I think this probably explains cover shooters with enemies that wait patiently for you to kill them as well. To delve further into tinfoil hat territory, I wouldn't be surprised if it came to light that hard modes were frequently dumbed down to encourage trophy/cheev hunters. Hasn't it long been understood that easier achievements lead to increased sales?
I doubt most people actually play games for the sake of playing games. Anecdotally, everyone I know who plays them does so for the window dressings. The story, the juvenile power fantasies, the old-time skiffy racism. That's why that piece of shit Bioshock Infinite was scoring perfect 10s even though the nicest thing I'd seen anyone say about its mechanics was that they're "serviceable." When reviewers bothered to mention gameplay at all, that is.
The only way we'll see meaningful improvements in gameplay, especially challenging gameplay, is if the market suddenly decides it cares more about being challenged or having fun with the mechanics themselves than seeing trophies pop up, or getting to the next cut scene, or feeling like an armchair badass, and somehow I don't see that happening.
Oh well. There's always multiplayer.