It's a really difficult thing to generalize about. I think that what we need to take into account are three things:
1.) Many mainstream titles offer multiple difficulties to allow newcomers to feel welcome but allow Veterans (see what I did there?) to get a challenge.
2.) Even on higher difficulties, games are seen in a different light as they were when Pacman was released. For something like Gears of War, beating the campaign isn't impossible: sure, it's difficult, but it has a clear end, and a clear set of goals. Meanwhile, Pacman, with no goals and no end, seems like an insurmountable challenge. (Unless you break the game.)
3.) Many developers are much less focused on keeping the game challenging in favor of other elements: powerful storytelling, engaging gameplay, et cetera. And personally, I'm okay with that.
/WallOfText