The article below contains personal commentary on several advancements in player motion capture technology stretching from late 2007 to the present, as well as several personal discoveries derived from those advancements.
--------------------------------------------------------------------------------------
Head Tracking for Desktop VR Displays using the Wii Remote
--------------------------------------------------------------------------------------
Please watch this video before continuing... [http://www.youtube.com/watch?v=Jd3-eiid-Uw]
There are some limitations to the method displayed in the above video.
Johnny Lee's method, which I call "simple head tracking", allows the player to lean, crouch, and even jump as buttonless inputs. However, it can not distinguish where on screen the player's head is pointing. This can be proven in a normal remote-in-hand setup by going to the sensitivity settings [http://www.nintendo.com/consumer/systems/wii/en_na/settingsSensorBar.jsp#sensitivity] in the Wii Menu and changing the angle at which the Wii remote sees the sensor bar. (say, 45 degrees to the left) You'll notice that the two IRLEDs on the sensor bar (from the perspective of the Wii remote) appear to be closer together, which it interprets as the player being further away from the screen than when you were standing the same distance directly in front of the sensor bar. It can't tell the difference. For the same reason, simple head tracking also can not distinguish looking up from looking down. The Wii remote can only see the X-Y coordinates of the two IRLEDs, not which direction they're pointing.
------------------------------------------------
Headtracking (6DOF) with Wiimote
------------------------------------------------
A modification of Johnny Lee's method was discovered over month before E3 2008 by Manuel Unternaehrer which solves the problems I mentioned previously, creating a true here [http://en.wikipedia.org/wiki/6DOF].
Please watch this video before continuing... [http://www.youtube.com/watch?v=QZFhXjlIiXo]
(I realize that the person is sitting down in the video but he just as easily could of been standing up in front of a TV.)
How does it work?
The Wii remote depends on the positions of two IRLEDs relative to eachother in order to perceive depth by measuring the apparent distance it sees between them. A third IRLED, positioned to make a triangle, will move differently when looking and turning than it would if it were moving in distance from the screen. For example, if you look down, the IRLED in the middle would move closer to and pass through the "line" between the other two IRLEDs. Likewise, If you look left or right, the middle IRLED would appear to "lean" closer to one of the other two LEDs.
You can demonstrate this to yourself like so: Hold up three fingers, with the middle one closer to you than the other two. Then bend your wrist towards you, (to simulate "looking down") away from you, (to simulate "looking up") and twist it left and right. (to simulate "turning left and right") Your fingertips positions as they would be seen in 2D act exactly the same as the three IRLEDs on the cap. (the Wii remote's IR camera can only see in 2D)
Our eyes see the triangle made up of the three fingertips as a 3D object. Because our brain knows the exact size of the 3D object, it automatically calculates its exact position in 3D space. Games can do the same thing. The calculations aren't as complex as you might think.
What does the third IRLED give us?
With this head tracker, not only can you do everything Johnny Lee's original program could do, but you can also look and turn. In effect, the head tracker becomes its own pointer, and the "look margin" from Metroid Prime 3 that was previously bound to the handheld Wiimote can be reassigned to the head tracker. Because the handheld Wiimote is no longer bound to a "look margin", it is free to be pointed anywhere on screen.
-------------------------------------------------------
Wii MotionPlus and AiLive's LiveMove 2
-------------------------------------------------------
"While the promise of 1:1 Wii Remote control is a tantalising one in theory, so far we've seen it working on...Wii Sports 2. Uh, thanks. With the company that's supposed to bring smiles doing nothing of the sort, it's instead left to the MotionPlus' creator, boring tech company AiLive, to show us a video that, despite being shot in 10 minutes with one of the techs and a shitty handycam, really gets us all excited about the add-ons potential. You'll want to skip to around the 1:50 mark." -- Luke Plunkett, Kotaku [http://kotaku.com/5031301/this-is-how-to-get-us-excited-about-motionplus-nintendo]
Please watch this video before continuing... [http://www.youtube.com/watch?v=acND4sO3pJs]
As you can see, the Wii MotionPlus is fully capable of doing everything it claimed it could do.
-----------------------------------------------------
Wii MotionPlus + 6DOF Head Tracking
-----------------------------------------------------
Let's recap.
Simple Head Tracking
As-is, Johnny Lee's simple head tracker can allow the player to lean, crouch, and jump without using any buttons or joysticks. In his setup, you still cannot look (up/down) or turn. (left/right) Even if you bind the handheld Wiimote to a "look margin" similar to the one used in Metroid Prime 3, there isn't enough practical function in it to convince developers to integrate head tracking into their games. With just leaning, crouching, and jumping, it's just not worth requiring players to buy additional hardware, even if it's just another Wiimote and a pair of infrared LED glasses.
6DOF Head Tracking
With Manuel Unternaehrer's 6DOF head tracker, not only can you do everything Johnny Lee's original program could do, but you can also look and turn. In effect, the head tracker becomes its own pointer, and the "look margin" from Metroid Prime 3 that was previously bound to the handheld Wiimote can be reassigned to the head tracker. Because the handheld Wiimote is no longer bound to a "look margin", it is free to be pointed anywhere on screen. Apparently however, this still doesn't offer enough immersion to warrant much attention.
Wii MotionPlus
Then the Wii MotionPlus was announced. Combine the hands-free pointer of the 6DOF head tracker with the one-to-one motion tracking of the Wii MotionPlus, and you get an unbound Wiimote that you can point not just anywhere on-screen, but anywhere off-screen as well. Imagine the level of immersion you could get out of that system. For example, if you know there's an enemy behind you, you can point the Wiimote over your shoulder and fire. Or if you know there's an enemy right outside your peripheral vision you could fire in that direction instead of turning to look. (without head tracking, you wouldn't be able to do this because the wiimote would have to be pointing at the screen at all times in order to look and turn)
"Twomote"
There are several options now available to us. If we only need bindings for moving and strafing, (because looking and turning is assigned to the head tracking "pointer") we could technically bind those functions to a Wiimote's D-pad. If we do that, then we don't need the nunchuck anymore, and can finally start using two Wiimotes, one in each hand. (which I call "Twomote" Style)
Balance Board Centering
Until the price of the Wii Balance Board goes down, it is my opinion that developers should have this hardware be completely optional. Once the price drops sufficiently though, it is possible that developers could expect the player to use the Balance Board not just for its inherent features, but also as a "default" spot on the floor for the player to return to for centering. (really though, if the inherent features aren't used, players would be better off using something else to mark their "default" spot)
Head Tracking Kit
The cost to manufacture, market, and buy the necessary additional hardware required for head tracking -- namely the IRLED headgear and wiimote stand -- can be minimized. As you can see at the end of Manuel Unternaehrer's 6DOF head tracking video [http://www.youtube.com/watch?v=QZFhXjlIiXo], the three IRLEDs are arranged around the rim of an ordinary baseball cap. Those LEDs could potentially be attached to clips, meaning Nintendo wouldn't have to sell goggles or glasses, they could just sell the 3 IRLED clip set and ask the player to attach them to a baseball cap they already own. (Nintendo could also sell their own caps separately or bundled without having to modify the cap in the slightest)
As for the wiimote stand, I personally have designed and successfully tested a wiimote stand made entirely out of posterboard which features simple construction, four pre-defined angles, and supports the weight of the wiimote beautifully.
Obviously, a commercial version would probably use laminated cardstock or something stronger but just as cheap, and use tabs instead of tape to hold it together.
-----------------------------------
Nintendo's Mixed Signals
-----------------------------------
The two devices I described previously are cheap to produce and sell, so why hasn't Nintendo announced head tracking yet? Emailing them for comment on the discovery of IRLED head tracking prompted the expected self-promoting response:
Greetings,
At this time, there has not been [an] announcement concerning using such a feature. As the worldwide leader and innovator in the creation of interactive entertainment, and because of our history of pushing the limits of gameplay and design, you can always count on Nintendo offering the most imaginative products.
In the meantime, keep checking our website's "What's New" section (http://www.nintendo.com/whatsnew [http://www.nintendo.com/whatsnew]) for the latest information and announcements.
Sincerely,
Dale Thompson
Nintendo of America Inc.As we've seen with the WiiMot+, Nintendo's proven itself to be overly-secretive with its hardware development. It may be that Nintendo is well aware of IRLED head tracking and is working on such a feature, but Johnny Lee's comments on the removal of the Boom Blox head tracking easter egg imply the opposite:"I have been talking to some Wii game developers and they've said that if [ a game ] requires too much movement on the player's part, Nintendo asks them to pull it." -- Johnny Lee, Interview with Nintendo Wii Fanboy [http://www.nintendowiifanboy.com/2008/04/30/lee-we-may-see-head-tracking-from-sony-or-microsoft-first/]I understand to a degree why Nintendo may be limiting player motion, but head tracking is -- as Darwin would say -- the next logical step after coming down from the trees immersion-wise. Gaming culture should at the very least be ready to get up off the couch and play, (at least that's what Nintendo would have us believe with games like Wii Fit) so what's with the mixed signals?
As I see it, this is where gaming is going and if Nintendo doesn't get on the ball with head tracking then Microsoft and Sony will. (Technically, Sony already has [http://kotaku.com/351539/vr-head-tracking-for-the-ps3], and already has the "stand" peripheral covered) Personally, I'd prefer Nintendo because the Wii remote although costing about the same as a PS3 Eye has more functionality besides its IR camera, meaning that you could already own an extra Wii remote and just use that or buy one for the head tracking and as an added bonus use it as a controller.
----------------------
Tracking Hands
----------------------
Currently, the Wii MotionPlus (WiiMot+) is capable of tracking its own movement in 3D space one-to-one. However, by conventional means it can only track this movement relative to the location it was at when the WiiMot+ was powered on. That is, if you held the WiiMot+ three feet above the ground, turned it on, made a motion, turned it off, turned it on again a foot higher and made the same motion, the WiiMot+ would not be able to tell the difference.
With 6DOF head tracking though, it is possible to tell the WiiMot+ exactly where it is located in 3D space regardless of when and where you turn the device on.
As you're aware from earlier, a 3IRLED 6DOF head tracker is capable of determining the exact position and orientation of the player's head in 3D space. Since the system knows the location of your head relative to the head tracker, it's possible to tell the WiiMot+ in-hand where its position in 3D space is by virtue of the fact that the in-hand Wii remote's IR camera has the same capabilities as the head tracker.
It's all a matter of relative perspective. As we know, the reason IR head tracking works at all is because conventionally the Wii remote detects the stationary sensor bar's IRLEDs as "moving" according to its own perspective and discerns where it's pointing from that.
The above image shows a stationary head tracking Wii remote and an in-hand Wii remote equipped with the WiiMot+. Both are pointing at a 3IRLED hat. The two boxes shown below the Wii remotes represent the approximate positions of the hat's three IRLEDs from the perspective of the corresponding camera. Because the system knows where your head is in 3D space courtesy of the head tracking Wii remote, any in-hand Wii remote can see the three IRLEDs on your hat and make the same calculations as the stationary head tracker to determine the location of your head in 3D space relative to that in-hand Wii remote. Since that in-hand Wii remote now knows your head's location relative to itself, it can determine it's own location relative to your head.
NOTE: The in-hand Wii remote only needs to be pointed at the IR hat during calibration.
Presto! The in-hand MotionPlus-equipped Wii remote now knows an initial position to start measuring one-to-one motion from. Naturally, this method can also be applied to "twomote".
------------------------------------------------
Mapping Yourself and your Screen
------------------------------------------------
Mapping the Screen
A while back, I discovered that a method for increasing pointer accuracy used in the setting screens for the Wii version of Zelda: Twilight Princess could be used to tell the game where your screen is in 3D space relative to the sensor bar. Then, if the system knew where the sensor bar was in 3D space relative to the Wii remote, it would know where the screen is relative to the Wii remote.
However, I later realized there's a much easier way to tell the system where your TV is in 3D space. All you need to do is touch at least three corners of the TV with the tip of a Wii MotionPlus-equipped Wii remote. (WiiMot+) The accelerometer in the Wiimote could even be used to automatically detect when it touches an object.
NOTE: Once the system knows where the screen is in 3D space relative to the WiiMot+, the system becomes capable of knowing when the WiiMot+ is pointing at the screen and where, without using the sensor bar. (although the sensor bar would probably still be useful for verifying the pointer's location)
Mapping Yourself
In-game avatars are relatively inaccurate. Although you can customize the face on your Mii (Wii), Homeboy (PS3), and Avatars (360), everything below the chin is disproportionate to your real body. Guess what? It's possible to tell the system your body proportions by using the WiiMot+. Once it knows your body proportions, it's possible for the system to render your entire body virtually. Useful proportions might include such measurements as: how high your waist is above ground, the width of your shoulders, and the distance between your shoulder and your hand.
How to Build Your Ragdoll
The method for acquiring your body proportions is similar to that used to find the location of the screen. I'm going to assume for these instructions that the player is holding the Wii MotionPlus-equipped Wiimote in their right hand.
First, you place the WiiMot+ on the ground to give it the height of the floor. Then, you pick it up and stand up straight. Next, you hold the WiiMot+ to the side of your waist and level with the ground to give it the distance between your waist and your feet. Then you give it your shoulder height by tapping your left shoulder with the tip of the WiiMot+ held in your right hand.
It's important that you tap the shoulder opposite the hand holding the WiiMot+ because the last step is the hold that arm straight down against your side. This diagonal motion has horizontal and vertical displacement. The horizontal displacement gives the system the width of your shoulders, and the vertical displacement gives the length of your arm.
Tracking the Rest of Your Body
You're probably wondering how the system's supposed to know where every other part of your body is. We can safely assume you won't hold yourself in an uncomfortable position while playing. Because we can assume this, we can also assume there's only one position the body can be in when the hands and head are stationary. (Sure, you might be able to move around a little, but not enough to affect gameplay.)
We can also assume that we don't want to track the exact physical location of the player's legs. Although it is possible to do so using something like a DDR pad, most people don't want to accidentally kick-in their TV screen. It's acceptable to simply know how much the knees are bending. (If your head moves any lower than the height of a standing position, it knows your knees are bent.)
On the systems end, from my perspective it's unlikely that these motion capture calculations would be particularly difficult. All they'd have to do is take an existing ragdoll 3D model and force the ragdoll to move it's hands and head to match your own hand and head movements.
For example: If the system knows how long your arm is, if you move an in-hand WiiMot+ any closer than that it knows that your bending your elbow. Because you're human, it also knows exactly how much you're bending your elbow.
------------------------
All Together Now
------------------------
Let's recap one more time:
Johnny Lee's simple headtracker introduced the idea of Wii head tracking, but it was incomplete. It could not determine the exact physical location of the player's head for the same reasons a normal remote-in-hand setup can't determine the exact location of a stationary sensor bar.
Michael Unternaehrer discovered several months later that a third dot was necessary to fix this issue. A third point causes the input for changes in yaw to look different from changes in distance, and the input for changes in pitch to look different from changes in height. Now, the head tracker can determine the exact location of the player's head.
Nintendo announced the Wii MotionPlus attachment at the less-than-optimal E3 2008. Although there was still speculation as to if the WiiMot+ could do what it promised, we later received proof from AiLive that the Wii MotionPlus is fully capable of one-to-one motion tracking.
Using a WiiMot+ in each hand (i.e. "Twomote") would allow the system to know the exact location of the player's hands. The Wii Balance Board could be used for specific in-game activities such as tightrope-walking, surfboarding, skiing, etc. to make the gameplay even more immersive.
Nintendo is sending mixed signals about head tracking. On one end, we have games like WiiFit where the whole point is to get up and exercise, and on the other end Nintendo seems to be quashing how much movement a game can ask for.
Normally, the Wii MotionPlus is only capable of one-to-one motion tracking relative to the point in space where it was turned on. However, it's possible with the help of 6DOF head tracking to let it know exactly where it is relative to the rest of the system.
Finally, the WiiMot+ can be used to tell the system the location and size of physical objects. The sensor bar can be made obsolete by pointing out the location of the television screen, and a very precise ragdoll avatar can be built by using the WiiMot+ to give the system the player's body proportions.
There you have it.
The Wii is capable of full-blown motion capture, and it doesn't need any special hardware upgrades to do it. It does however need three Wii remotes, two Wii MotionPlus attachments, a Wii remote stand, a 3IRLED clip set, and a baseball cap.
That might sound expensive, but it's not. Most Wii owners already have three or four Wii remotes, so that's not too much to ask for. It's also not unreasonable to ask the player to have two Wii MotionPlus attachments, as WiiSports Resort requires two in order for two people to play, and everyone's going to want that game. Almost everyone these days has a baseball cap of some kind, and almost all baseball caps are shaped exactly the same way. So that's no additional cost so far.
The 3IRLED clip set and Wii remote stand are ridiculously cheap to manufacture, which results in them being ridiculously cheap to sell retail. You could probably buy both products together for less than $5. They're also simple enough that you could always make your own from scratch.
Full Motion
Never before in a home console system have we seen this level of immersion potential. Current consoles (particularly the Wii) using Player Motion Capture (which I call "Full Motion" for short) can make functions such as leaning, looking, turning, crouching, and jumping "buttonless", opening up those buttons for other functions. Joysticks may start to vanish from future consoles because they've been replaced with buttonless functions and motion controls.
Currently, this setup would require that movement controls be bound to a D-pad, but it's possible that the Wii may get a joystick attachment that sits on top of the D-pad, replacing it. The nunchuck doesn't get the one-to-one motion tracking of the Wii MotionPlus, so such an attachment could make the nunchuck controller obsolete.
If for some reason you were of the opinion that the Wii doesn't count as a "next gen" system, I hope this persuades you otherwise.
--------------------------------------------------------------------------------------
Head Tracking for Desktop VR Displays using the Wii Remote
--------------------------------------------------------------------------------------
Please watch this video before continuing... [http://www.youtube.com/watch?v=Jd3-eiid-Uw]
There are some limitations to the method displayed in the above video.
Johnny Lee's method, which I call "simple head tracking", allows the player to lean, crouch, and even jump as buttonless inputs. However, it can not distinguish where on screen the player's head is pointing. This can be proven in a normal remote-in-hand setup by going to the sensitivity settings [http://www.nintendo.com/consumer/systems/wii/en_na/settingsSensorBar.jsp#sensitivity] in the Wii Menu and changing the angle at which the Wii remote sees the sensor bar. (say, 45 degrees to the left) You'll notice that the two IRLEDs on the sensor bar (from the perspective of the Wii remote) appear to be closer together, which it interprets as the player being further away from the screen than when you were standing the same distance directly in front of the sensor bar. It can't tell the difference. For the same reason, simple head tracking also can not distinguish looking up from looking down. The Wii remote can only see the X-Y coordinates of the two IRLEDs, not which direction they're pointing.
------------------------------------------------
Headtracking (6DOF) with Wiimote
------------------------------------------------
A modification of Johnny Lee's method was discovered over month before E3 2008 by Manuel Unternaehrer which solves the problems I mentioned previously, creating a true here [http://en.wikipedia.org/wiki/6DOF].
Please watch this video before continuing... [http://www.youtube.com/watch?v=QZFhXjlIiXo]
(I realize that the person is sitting down in the video but he just as easily could of been standing up in front of a TV.)
How does it work?
The Wii remote depends on the positions of two IRLEDs relative to eachother in order to perceive depth by measuring the apparent distance it sees between them. A third IRLED, positioned to make a triangle, will move differently when looking and turning than it would if it were moving in distance from the screen. For example, if you look down, the IRLED in the middle would move closer to and pass through the "line" between the other two IRLEDs. Likewise, If you look left or right, the middle IRLED would appear to "lean" closer to one of the other two LEDs.
You can demonstrate this to yourself like so: Hold up three fingers, with the middle one closer to you than the other two. Then bend your wrist towards you, (to simulate "looking down") away from you, (to simulate "looking up") and twist it left and right. (to simulate "turning left and right") Your fingertips positions as they would be seen in 2D act exactly the same as the three IRLEDs on the cap. (the Wii remote's IR camera can only see in 2D)
Our eyes see the triangle made up of the three fingertips as a 3D object. Because our brain knows the exact size of the 3D object, it automatically calculates its exact position in 3D space. Games can do the same thing. The calculations aren't as complex as you might think.
What does the third IRLED give us?
With this head tracker, not only can you do everything Johnny Lee's original program could do, but you can also look and turn. In effect, the head tracker becomes its own pointer, and the "look margin" from Metroid Prime 3 that was previously bound to the handheld Wiimote can be reassigned to the head tracker. Because the handheld Wiimote is no longer bound to a "look margin", it is free to be pointed anywhere on screen.
-------------------------------------------------------
Wii MotionPlus and AiLive's LiveMove 2
-------------------------------------------------------
"While the promise of 1:1 Wii Remote control is a tantalising one in theory, so far we've seen it working on...Wii Sports 2. Uh, thanks. With the company that's supposed to bring smiles doing nothing of the sort, it's instead left to the MotionPlus' creator, boring tech company AiLive, to show us a video that, despite being shot in 10 minutes with one of the techs and a shitty handycam, really gets us all excited about the add-ons potential. You'll want to skip to around the 1:50 mark." -- Luke Plunkett, Kotaku [http://kotaku.com/5031301/this-is-how-to-get-us-excited-about-motionplus-nintendo]
Please watch this video before continuing... [http://www.youtube.com/watch?v=acND4sO3pJs]
As you can see, the Wii MotionPlus is fully capable of doing everything it claimed it could do.
-----------------------------------------------------
Wii MotionPlus + 6DOF Head Tracking
-----------------------------------------------------
Let's recap.
Simple Head Tracking
As-is, Johnny Lee's simple head tracker can allow the player to lean, crouch, and jump without using any buttons or joysticks. In his setup, you still cannot look (up/down) or turn. (left/right) Even if you bind the handheld Wiimote to a "look margin" similar to the one used in Metroid Prime 3, there isn't enough practical function in it to convince developers to integrate head tracking into their games. With just leaning, crouching, and jumping, it's just not worth requiring players to buy additional hardware, even if it's just another Wiimote and a pair of infrared LED glasses.
6DOF Head Tracking
With Manuel Unternaehrer's 6DOF head tracker, not only can you do everything Johnny Lee's original program could do, but you can also look and turn. In effect, the head tracker becomes its own pointer, and the "look margin" from Metroid Prime 3 that was previously bound to the handheld Wiimote can be reassigned to the head tracker. Because the handheld Wiimote is no longer bound to a "look margin", it is free to be pointed anywhere on screen. Apparently however, this still doesn't offer enough immersion to warrant much attention.
Wii MotionPlus
Then the Wii MotionPlus was announced. Combine the hands-free pointer of the 6DOF head tracker with the one-to-one motion tracking of the Wii MotionPlus, and you get an unbound Wiimote that you can point not just anywhere on-screen, but anywhere off-screen as well. Imagine the level of immersion you could get out of that system. For example, if you know there's an enemy behind you, you can point the Wiimote over your shoulder and fire. Or if you know there's an enemy right outside your peripheral vision you could fire in that direction instead of turning to look. (without head tracking, you wouldn't be able to do this because the wiimote would have to be pointing at the screen at all times in order to look and turn)
"Twomote"
There are several options now available to us. If we only need bindings for moving and strafing, (because looking and turning is assigned to the head tracking "pointer") we could technically bind those functions to a Wiimote's D-pad. If we do that, then we don't need the nunchuck anymore, and can finally start using two Wiimotes, one in each hand. (which I call "Twomote" Style)
Balance Board Centering
Until the price of the Wii Balance Board goes down, it is my opinion that developers should have this hardware be completely optional. Once the price drops sufficiently though, it is possible that developers could expect the player to use the Balance Board not just for its inherent features, but also as a "default" spot on the floor for the player to return to for centering. (really though, if the inherent features aren't used, players would be better off using something else to mark their "default" spot)
Head Tracking Kit
The cost to manufacture, market, and buy the necessary additional hardware required for head tracking -- namely the IRLED headgear and wiimote stand -- can be minimized. As you can see at the end of Manuel Unternaehrer's 6DOF head tracking video [http://www.youtube.com/watch?v=QZFhXjlIiXo], the three IRLEDs are arranged around the rim of an ordinary baseball cap. Those LEDs could potentially be attached to clips, meaning Nintendo wouldn't have to sell goggles or glasses, they could just sell the 3 IRLED clip set and ask the player to attach them to a baseball cap they already own. (Nintendo could also sell their own caps separately or bundled without having to modify the cap in the slightest)
As for the wiimote stand, I personally have designed and successfully tested a wiimote stand made entirely out of posterboard which features simple construction, four pre-defined angles, and supports the weight of the wiimote beautifully.
Obviously, a commercial version would probably use laminated cardstock or something stronger but just as cheap, and use tabs instead of tape to hold it together.
-----------------------------------
Nintendo's Mixed Signals
-----------------------------------
The two devices I described previously are cheap to produce and sell, so why hasn't Nintendo announced head tracking yet? Emailing them for comment on the discovery of IRLED head tracking prompted the expected self-promoting response:
Greetings,
At this time, there has not been [an] announcement concerning using such a feature. As the worldwide leader and innovator in the creation of interactive entertainment, and because of our history of pushing the limits of gameplay and design, you can always count on Nintendo offering the most imaginative products.
In the meantime, keep checking our website's "What's New" section (http://www.nintendo.com/whatsnew [http://www.nintendo.com/whatsnew]) for the latest information and announcements.
Sincerely,
Dale Thompson
Nintendo of America Inc.As we've seen with the WiiMot+, Nintendo's proven itself to be overly-secretive with its hardware development. It may be that Nintendo is well aware of IRLED head tracking and is working on such a feature, but Johnny Lee's comments on the removal of the Boom Blox head tracking easter egg imply the opposite:"I have been talking to some Wii game developers and they've said that if [ a game ] requires too much movement on the player's part, Nintendo asks them to pull it." -- Johnny Lee, Interview with Nintendo Wii Fanboy [http://www.nintendowiifanboy.com/2008/04/30/lee-we-may-see-head-tracking-from-sony-or-microsoft-first/]I understand to a degree why Nintendo may be limiting player motion, but head tracking is -- as Darwin would say -- the next logical step after coming down from the trees immersion-wise. Gaming culture should at the very least be ready to get up off the couch and play, (at least that's what Nintendo would have us believe with games like Wii Fit) so what's with the mixed signals?
As I see it, this is where gaming is going and if Nintendo doesn't get on the ball with head tracking then Microsoft and Sony will. (Technically, Sony already has [http://kotaku.com/351539/vr-head-tracking-for-the-ps3], and already has the "stand" peripheral covered) Personally, I'd prefer Nintendo because the Wii remote although costing about the same as a PS3 Eye has more functionality besides its IR camera, meaning that you could already own an extra Wii remote and just use that or buy one for the head tracking and as an added bonus use it as a controller.
----------------------
Tracking Hands
----------------------
Currently, the Wii MotionPlus (WiiMot+) is capable of tracking its own movement in 3D space one-to-one. However, by conventional means it can only track this movement relative to the location it was at when the WiiMot+ was powered on. That is, if you held the WiiMot+ three feet above the ground, turned it on, made a motion, turned it off, turned it on again a foot higher and made the same motion, the WiiMot+ would not be able to tell the difference.
With 6DOF head tracking though, it is possible to tell the WiiMot+ exactly where it is located in 3D space regardless of when and where you turn the device on.
As you're aware from earlier, a 3IRLED 6DOF head tracker is capable of determining the exact position and orientation of the player's head in 3D space. Since the system knows the location of your head relative to the head tracker, it's possible to tell the WiiMot+ in-hand where its position in 3D space is by virtue of the fact that the in-hand Wii remote's IR camera has the same capabilities as the head tracker.
It's all a matter of relative perspective. As we know, the reason IR head tracking works at all is because conventionally the Wii remote detects the stationary sensor bar's IRLEDs as "moving" according to its own perspective and discerns where it's pointing from that.
The above image shows a stationary head tracking Wii remote and an in-hand Wii remote equipped with the WiiMot+. Both are pointing at a 3IRLED hat. The two boxes shown below the Wii remotes represent the approximate positions of the hat's three IRLEDs from the perspective of the corresponding camera. Because the system knows where your head is in 3D space courtesy of the head tracking Wii remote, any in-hand Wii remote can see the three IRLEDs on your hat and make the same calculations as the stationary head tracker to determine the location of your head in 3D space relative to that in-hand Wii remote. Since that in-hand Wii remote now knows your head's location relative to itself, it can determine it's own location relative to your head.
NOTE: The in-hand Wii remote only needs to be pointed at the IR hat during calibration.
Presto! The in-hand MotionPlus-equipped Wii remote now knows an initial position to start measuring one-to-one motion from. Naturally, this method can also be applied to "twomote".
------------------------------------------------
Mapping Yourself and your Screen
------------------------------------------------
Mapping the Screen
A while back, I discovered that a method for increasing pointer accuracy used in the setting screens for the Wii version of Zelda: Twilight Princess could be used to tell the game where your screen is in 3D space relative to the sensor bar. Then, if the system knew where the sensor bar was in 3D space relative to the Wii remote, it would know where the screen is relative to the Wii remote.
However, I later realized there's a much easier way to tell the system where your TV is in 3D space. All you need to do is touch at least three corners of the TV with the tip of a Wii MotionPlus-equipped Wii remote. (WiiMot+) The accelerometer in the Wiimote could even be used to automatically detect when it touches an object.
NOTE: Once the system knows where the screen is in 3D space relative to the WiiMot+, the system becomes capable of knowing when the WiiMot+ is pointing at the screen and where, without using the sensor bar. (although the sensor bar would probably still be useful for verifying the pointer's location)
Mapping Yourself
In-game avatars are relatively inaccurate. Although you can customize the face on your Mii (Wii), Homeboy (PS3), and Avatars (360), everything below the chin is disproportionate to your real body. Guess what? It's possible to tell the system your body proportions by using the WiiMot+. Once it knows your body proportions, it's possible for the system to render your entire body virtually. Useful proportions might include such measurements as: how high your waist is above ground, the width of your shoulders, and the distance between your shoulder and your hand.
How to Build Your Ragdoll
The method for acquiring your body proportions is similar to that used to find the location of the screen. I'm going to assume for these instructions that the player is holding the Wii MotionPlus-equipped Wiimote in their right hand.
First, you place the WiiMot+ on the ground to give it the height of the floor. Then, you pick it up and stand up straight. Next, you hold the WiiMot+ to the side of your waist and level with the ground to give it the distance between your waist and your feet. Then you give it your shoulder height by tapping your left shoulder with the tip of the WiiMot+ held in your right hand.
It's important that you tap the shoulder opposite the hand holding the WiiMot+ because the last step is the hold that arm straight down against your side. This diagonal motion has horizontal and vertical displacement. The horizontal displacement gives the system the width of your shoulders, and the vertical displacement gives the length of your arm.
Tracking the Rest of Your Body
You're probably wondering how the system's supposed to know where every other part of your body is. We can safely assume you won't hold yourself in an uncomfortable position while playing. Because we can assume this, we can also assume there's only one position the body can be in when the hands and head are stationary. (Sure, you might be able to move around a little, but not enough to affect gameplay.)
We can also assume that we don't want to track the exact physical location of the player's legs. Although it is possible to do so using something like a DDR pad, most people don't want to accidentally kick-in their TV screen. It's acceptable to simply know how much the knees are bending. (If your head moves any lower than the height of a standing position, it knows your knees are bent.)
On the systems end, from my perspective it's unlikely that these motion capture calculations would be particularly difficult. All they'd have to do is take an existing ragdoll 3D model and force the ragdoll to move it's hands and head to match your own hand and head movements.
For example: If the system knows how long your arm is, if you move an in-hand WiiMot+ any closer than that it knows that your bending your elbow. Because you're human, it also knows exactly how much you're bending your elbow.
------------------------
All Together Now
------------------------
Let's recap one more time:
Johnny Lee's simple headtracker introduced the idea of Wii head tracking, but it was incomplete. It could not determine the exact physical location of the player's head for the same reasons a normal remote-in-hand setup can't determine the exact location of a stationary sensor bar.
Michael Unternaehrer discovered several months later that a third dot was necessary to fix this issue. A third point causes the input for changes in yaw to look different from changes in distance, and the input for changes in pitch to look different from changes in height. Now, the head tracker can determine the exact location of the player's head.
Nintendo announced the Wii MotionPlus attachment at the less-than-optimal E3 2008. Although there was still speculation as to if the WiiMot+ could do what it promised, we later received proof from AiLive that the Wii MotionPlus is fully capable of one-to-one motion tracking.
Using a WiiMot+ in each hand (i.e. "Twomote") would allow the system to know the exact location of the player's hands. The Wii Balance Board could be used for specific in-game activities such as tightrope-walking, surfboarding, skiing, etc. to make the gameplay even more immersive.
Nintendo is sending mixed signals about head tracking. On one end, we have games like WiiFit where the whole point is to get up and exercise, and on the other end Nintendo seems to be quashing how much movement a game can ask for.
Normally, the Wii MotionPlus is only capable of one-to-one motion tracking relative to the point in space where it was turned on. However, it's possible with the help of 6DOF head tracking to let it know exactly where it is relative to the rest of the system.
Finally, the WiiMot+ can be used to tell the system the location and size of physical objects. The sensor bar can be made obsolete by pointing out the location of the television screen, and a very precise ragdoll avatar can be built by using the WiiMot+ to give the system the player's body proportions.
There you have it.
The Wii is capable of full-blown motion capture, and it doesn't need any special hardware upgrades to do it. It does however need three Wii remotes, two Wii MotionPlus attachments, a Wii remote stand, a 3IRLED clip set, and a baseball cap.
That might sound expensive, but it's not. Most Wii owners already have three or four Wii remotes, so that's not too much to ask for. It's also not unreasonable to ask the player to have two Wii MotionPlus attachments, as WiiSports Resort requires two in order for two people to play, and everyone's going to want that game. Almost everyone these days has a baseball cap of some kind, and almost all baseball caps are shaped exactly the same way. So that's no additional cost so far.
The 3IRLED clip set and Wii remote stand are ridiculously cheap to manufacture, which results in them being ridiculously cheap to sell retail. You could probably buy both products together for less than $5. They're also simple enough that you could always make your own from scratch.
Full Motion
Never before in a home console system have we seen this level of immersion potential. Current consoles (particularly the Wii) using Player Motion Capture (which I call "Full Motion" for short) can make functions such as leaning, looking, turning, crouching, and jumping "buttonless", opening up those buttons for other functions. Joysticks may start to vanish from future consoles because they've been replaced with buttonless functions and motion controls.
Currently, this setup would require that movement controls be bound to a D-pad, but it's possible that the Wii may get a joystick attachment that sits on top of the D-pad, replacing it. The nunchuck doesn't get the one-to-one motion tracking of the Wii MotionPlus, so such an attachment could make the nunchuck controller obsolete.
If for some reason you were of the opinion that the Wii doesn't count as a "next gen" system, I hope this persuades you otherwise.
