Sixth Sense by Fluid Interface Group

The MIT Fluid Interface Group demonstrated a new application of Mobile Projective Augmented Reality (MPAR). They call it SixthSense. Their wearable system converts any surface into interactive display using a mobile nano-projector. The user interaction is done using colored markers on fingers. The system is connected to internet, and able to display real time information related to subject, for example, while reading a news paper, it can display related videos, or while searching for a book, it can display rating and related information from websites such as Amazon or Wikipidea. See the demo videos below:





Here are some of my initial comments after watching above video clips:

1. They don't use any pattern or object recognition, e.g. in newspaper demo, markers can be clearly seen to identify the region to display the video.
2. I doubt they use face recognition to identify the person... However, they are able to project correct geometry on planar as well as non-planar surfaces to some extent..
3. I dont believe they use real time information, e.g. in taxi, the information about airport is displayed on the boarding pass, moreover in this demo, the information on ticket is not read or recognized automatically. However, this could be very interesting application.
4. They did implement some marker based gesture recognition algorithm. However, number of gestures recognized are limited to specific application only.

In summary, its nothing special or new... They have just presented some potential applications of the existing technology. For these applications, a lot of research is required in these domains: pattern or object recognition, natural feature tracking, gesture recognition and data mining.