I think Yahtzee put it best in one of his columns when he said something to the effect of, "No technology that is less convenient to use than what came before it." At the time, he was criticizing either 3D or motion control, but it applies to VR I think (ironic, considering how he is enthused about VR).
I truly believe that, to the average consumer, the "wow" factor of VR (at least what's coming down the pipe now) is not enough to overcome the inconvenience of having to put on the headset, calibrate it, deal with motion sickness, etc, and that's not even getting into the cost. The number of people who need to buy it for it to be "the next big thing" won't bother with it, at least with this generation of the tech.
Motion control didn't have staying power because the core gaming audience just wanted to sit down on the couch or a chair, boot up a game on their TV or PC monitor (which they already own and serve many, MANY more purposes that just displaying games) and play. As soon as the novelty wore off among those who thought motion control was "neat," the motion control fell of the map.
3D didn't have staying power because of the (at the time) high-cost of 3D-capable TVs, the higher-cost of 3D movie tickets, the need to wear the (99% of the time uncomfortable) glasses, and the fact that due to physical conditions, many people couldn't see it right or experienced motion sickness.
As for VR? It has many of the drawbacks of both of the above. Like motion control, it's less convenient and more "involved" than the simple person-machine relationship we currently have. Like 3D, some people just can't use it and it's more expensive. A key difference this time around is that the audience that is "wowed" by VR is not the "casual gamers" and "non-gamers" that were wowed by the Wii. The VR enthusiast audience seems to be mostly core gamers and tech-heads.