the delicate art of noise pollution

This is part two of a two part article on the sixth sense device developed at MIT (you can read part one here). Before you read this article, please watch this video so you know what I am writing about (if you have seen it already, read on with swift abandon).

wax enthusiastically

With the concerns out of the way I can wax freely and enthusiastically about the wonders of the sixth sense device. Because wondrous it is. It is one of those ideas that has been thought of before by science fiction writers in one form or another but could not be put into practice due to sheer lack of processing power and constrictions of size. Which is nicely illustrated in the video when a giant projector is balanced precariously on the head of one of the developers.

cheap prototype

Technology has come far enough however that this device, as it is now, can be built for around 350 dollars (according to the site) and be worn around the neck as an oversized pendent. Prototypes have come a long way. Imagine what this could become as a product from a company that has the will and production power to make this as a mass product. It could become as small as a bluetooth headset of a very small pendent worn as jewellery. Smart jewellery!

extrapolated

The possibilities this technology offers are mind boggling. The video only lifts part of the veil that an active imagination could blow away altogether. Mind you, much of this is purely a writers imagination, heavily extrapolated on what I believe technology capable of in the near future but I will try and base it as much as possible on what I saw in the video. Lets explore!

the plumbers phone number

When worn by you the device records your surroundings. Apart from recording it also ‘knows’ what it sees. It can recognise discrete parts of its surroundings – as shown when text is lifted off a page. Combined with a now already almost ubiquitous Internet connection this means that everything around us, our whole environment becomes a web site. The plumbers phone number on the side of his van, the name of the bakery store on the front of the building, the display window of the electronics store. All these items can be picked out and searched for on the Internet with a gesture. Google has already begun implementing face recognition into its image search. When this technology gets widened to searching on any part of an image, we could combine the image of that desirable camera plucked out of the shop window with an Internet search for exactly that camera and get all the information we would want on it – including its best price – from just looking into a shop window. The shop (window) becomes a much more integral part of the web as it has already become thanks to barcode scanning. There is no distinction between the shop’s web site and the actual shop.

Strasbourg

Another example of how the physical world could have become a web site thanks to this device is a visit to the Historical Museum in Strasbourg, France a couple of years ago. We were travelling back from a wedding in France and after a detour through Switzerland we decided to stay a couple of days in the Alsace. We visited Strasbourg and discovered it as a vibrant and beautiful city. So to learn more about the history of this city we visited the Historical Museum where the history of this ancient city is displayed in clearly defined stages. What I saw exhibited there triggered many questions. Imagine if I could have just virtually grabbed pieces from the museum display and could have stored it for later perusal and putting into context. For instance when I later happened to listen to a BBC podcast on the Holy Roman Empire, which played an important part in the history of Strasbourg. I could have used information from the museum to augment the podcast. Again, the museum would have become a retrievable web site.

positive side of the coin

The copyright argument I made yesterday is the negative side of a coin and with that negative side out of the way let us look at the positive side of the same coin. The Internet is full of knowledge. The problem is that you sometimes just do not know where to begin your search for that knowledge. Also you sometimes do not know you want to know more about something until you are confronted with it in real life (as the museum example showed). With this device any information that only exists in the real world can be digitised and incorporated into the already huge pool of knowledge. There are many initiatives under way to digitise our analogue world. Google is digitising books at alarming speed, so is the Project Gutenburg and rumours have it, so is Apple. Yet partly because of copyright issues, partly because of the sheer vastness of the task, they can not digitise everything. And even if they did, you may come across something serendipitously. To be able to grab that moment and store it for later use is like browsing YouTube for that cat playing the piano and coming across some cool music you never heard before. With this device it becomes possible to browse your world and store bookmarks, interactive snapshots, of that world much as you do on the Internet.

serendipity

So real life serendipity becomes storable, retrievable and searchable and environment becomes a web site. What else could we imagine about this device?

user interface

A major part of the video shows us that our world becomes our user interface. Any surface will do. A hand palm to dial the numbers of a phone number, a wall to flick through photo’s and even edit them and the floor of a tram to play games on.

real world Mirror’s Edge

That gaming intrigues me. The Wii showed us that a game controller can become an extension of our movements. Microsoft and Sony are hard at work incorporating gestures into their gaming systems. But with the sixth sense device the game itself can become part of the world outside the console. We no longer need to be at or near the console, something even the Wii requires. We can bring our game to the street and play it there with parts of the environment becoming part of the game world. Although this might sound worrying in the case of ultra violent games like Modern Warfare 2 or Grand Theft Auto, for other games this might actually become fun. Ignoring the legalities for the moment, imagine a parkour game like Mirror’s Edge, where your real world environment becomes the obstacle course. Enemies could be projected on walls and surfaces and your dodging and weaving could make for an exhaustive bit of gaming.

questing

Another genre of game that could transfer to this device is adventure gaming. It would be the next step up from geocaching. Especially if you play your game in an atmosphere where an adventure game would fit best, say a dark forest. Anything in your environment could become part of a fictional world to explore. A simple stick could become a powerful magic sword to battle projected dragons with. World of Warcraft could begin incorporating questing with physical, real world friends.

re-define a stereotype

As a matter of fact the gaming potential of this piece of equipment is so huge that I could write a separate blog post about it. The main thing to take away from it is that if the Wii has liberated us from chewing crisps on the couch and grow a certain rotundity, this would be the next step. Imagine that: gamers as healthy, lean individuals, running about in the real world. We would have to re-define some stereotypes. Whatever next…

body language

The device could also change the way we communicate online. It would bring together all ways of communication we use now and then some. In my view, the main advantage of the device in communication would be that it would be possible to incorporate body language in online communication. The device could somehow translate our body language and incorporate it in the data stream. So even in situations where video conferencing is not possible or advisable, we could somehow use a translation of body language. Maybe the device would need a sensor that senses our stance and general attitude but I think that is possible. In the video a microphone is used to detect the position of a finger on paper (talk about thinking out of the box!), a similar technique could be used to detect our muscle tension. Those tensions could be translated into a body-language-sign-language that could be transmitted digitally. Like emoticons on steroids.

open source

It is becoming a rather long blog post. My imagination is constantly triggered by this device. I constantly think of new ways to incorporate it in our daily lives. And so do others I am sure. The beauty of this sixth sense project at MIT is that it is not developed in some closet surrounded by secrecy and NDA’s. This project is developed openly and as Pranav Mistry says at the end of the video: the software will become available as open source. As the device itself is apparently built from easily obtainable and affordable hardware, there is no stopping the parallel development of applications for this device by who ever wants to. The only limit to its use is our imagination. So get brain storming!

If this device triggers your imagination as much as it does mine or indeed more so, please feel free to add your thoughts in the comments. And I’d love to read some wacky ideas as well. After all, technology shouldn’t always be taken too seriously