hardcore game = gameplay > presentation ( a game that emphasizes gameplay over presentation)
mainstream game = presentation > gameplay ( a game that emphasizes presentation over gameplay)
Vanquish, Dark Souls, Demon Souls and Bayonetta are all perfect examples of hardcore games. A hardcore game isn't just a game that is "hard" it's a game that is skill based and technically deep that, while it may be accessible on a beginner level, is difficult to master with a steep learning curve.
Contrast a technically deep, single-player only TPS with a short campaign, Vanquish, to Gears and Uncharted which are shallow bullet sponge fests with no technical depth and tons of cinematics. The selling points of those games are their presentation--lavish production values and lengthy cinematics. Singe-player experiences of those games are really weak and they have to count on online multiplayer to add depth to the experience as their is little to the singe-player game beyond the inital 8 hour first play to complete them. The "hard" modes are harder, but arbitrarily so, and don't require mastery of complex technique.
To futher make the distinction, there can be such a thing as a 'mainstream hardcore game'. Street Fighter is an example. SF is an incredibly popular franchise with mainstream appeal but it retains the technical depth of gameplay and steep learning curve to mastery of a hardcore game where an expert player would destroy any button mashing beginner.
Hardcore gaming is dead existing only as a tiny niche segment of the overall market today. It died when arcades died and gaming went mainstream. The market for hardcore games is inherently smaller and rarely commercially viable today.
Hardcore gaming belonged to another era when gaming's dedicated core audience was socially withdrawn and obsessive geeks and nerds.
