Much has been said about Microsoft’s Project Natal since its announcement in the 2009 Electronic Entertainment Expo (E3) last June 1.  “Natal” was used as the code name of the project because natal means birth and Microsoft considers the latest Xbox 360 video game device as the next generation home entertainment system.  It utilizes a motion sensor device that will eliminate the need for a traditional controller and lets the gamer use his full body to interact with the device while gaming. The sensor will capture the gamer’s actions and feed corresponding electrical signals to the Xbox 360 console.   Gamers will also be able to speak into the sensor, which will then feed your commands into the screen.

So what are the technologies utilized that makes Project Natal different from the present breed of gaming devices?  Industry sources are saying that the technology behind Project Natal was based on ZCam, a time-of-flight camera product from 3DV Systems.  This company was bought by Microsoft in March 2009, just before the announcement of Project Natal in E3.  The software which runs the Zcam was in turn developed internally by Microsoft.  The software enables gesture recognition, facial recognition, and voice recognition.

The motion sensor device contains these components - a CMOS video camera, an infra red source, a multi-array microphone and a custom processor that utilizes a proprietary software for this application.  The video camera and infra red source are used to detect motion and measure distance.  The multi-array microphone allows the device to do voice recognition and acoustic localization.  There is nothing new in these technologies but the magic lies probably in the software that Microsoft developed for this application.  The following paragraphs provide an overview of the technologies used.

Gesture Recognition Interface

The gaming industry has seen developments in Gesture Recognition Interface or GRI in the past few years. This technology has gained popularity in gamers because it either complements or eliminates traditional gaming controllers.  The early models of GRI devices are based on 2D cameras.  But these devices did not gain very wide acceptance because they lack depth perception, which is very difficult to accomplish with ordinary 2D cameras.  Segmentation of a hand movement from any background can be done only on basis of color and motion only.  This requires heavy calibration and is affected by changes in lighting available in a particular scene.  This problem can be solved by incorporating a 3D imaging technology for the GRI.

3D Imaging technology

A light pulse is first generated from an infra red light source creating a light wall.  This wall hits the 3D objects in the scene and the reflections create an imprint of the objects shape. This imprint contains all the information that will be required for the reconstruction of the scene structure.  The reflected light wall carrying the imprint moves towards the camera carrying the image imprint.  But this camera does not work like an ordinary video camera.  It behaves in an on off manner as the imprint image enters the camera.  The result of this gating algorithm is a gray level image with each pixel's intensity proportional to the imprint image depth.  The depth resolution can be set by setting the depth measurement window width or the camera gating intervals.

Source: 3DV Systems

During the E3 event, several demos were conducted to show how the technologies discussed above were used.  One demo showed a chair being dragged into the middle of the playing field, and the Xbox 360 game device software was able to to track the human movement.  Another demo is a painting application, in which the entire body is used to handle virtual painting supplies, allowing the gamer to do things like throw buckets of paint onto a canvas.  The other demos included a skateboarding game with a lad "air" ollie-ing, and a family playing a car racing game, in which the dad was changing a tire while the daughter does the driving.  A standout demo was “Milo,” the modern day version of Sega’s Seaman (interactive character in the 1990s).  A conversation was conducted between a Microsoft employee and Milo, a virtual boy who communicates using the Project Natal sensor device.  A bit creepy but definitely a very cool stuff.

So when will the new Xbox 360 be out in the market? Microsoft has not announced yet a release target for Project Natal, but there are speculations that it will be out some time in 2010.  Some are even hoping that it will be available as early as the holiday season in December 2009.  Others are saying that it will be the killer app to the Wii.  Do you agree?