The virtual world listens to things and follows gestures

The virtual world listens to things and follows gestures

Whoever saw Spielberg’s “Minority Report” would hardly have guessed that the creators of this science-fiction film were filming an unknown to the general public and still crude (at that time) development, which had been carried out by American computer scientists for many years. True, the filming involved some special effects, but the concept was based on a real system.

The authors of the complex, which later evolved into G-Speak, are John Underkoffler and his colleagues from the MIT Media Lab. It was there that Spielberg and his subordinates glimpsed the current management of data using gestures.

Long before Minority Report, Underkoffler was thinking about how to bring computer graphics into the real world. He dreamed that photographs, diagrams, tables could be displayed on walls, floors, palms, tables and chairs.

Moreover, it would be good to manage these virtual objects with simple gestures. I drew a circle with my finger, cut out an object from the frame, swiped it palm down, moved the fragment to a new photo, made a few more passes, and sent the result by email. Something like this.

The conclusion was clear – we needed projectors. Moreover, scientists immediately thought about placing one or, better, several such installations on the ceiling and walls, where lighting bulbs are usually located. The light bulb has a “privileged position” in the room – it can clearly see everything.

The authors of the system joked that a light bulb is also in some way a projector, only showing one pixel. So all that was needed was to “raise the resolution”. That’s what American researchers did, building a very small digital projector (with the help of a number of industrial partners).

Gesture perception turned out to be a more difficult task. It was decided to build a tiny camera into the same “bulb-projector”. The camera would sense hand movements (wearing special gloves equipped with clearly visible markers) and transmit them to the computer. Then it’s a matter of technology: the PC must recognize gestures on the fly, determine the coordinates of the hands in space and interpret them as commands. The creation of such software, however, took a lot of time. It only looks simple in words.

Several of these “light bulbs” covering various “firing angles” in the room would create the environment that Underkoffler dreamed of. The authors even came up with the funny term “input-output lightbulb,” although this device has nothing to do with the light bulb.

This is how the Luminous Room appeared at the Massachusetts Institute in the 1990s. In it, Underkoffler spent many years polishing the principle of “aerial control of graphics,” and in it, members of Spielberg’s team spent whole days on end, so that they could then most plausibly show the computer interface of the “distant future” in the film. John Underkoffler, by the way, became a scientific consultant for Minority Report. We also owe him the thoughtfulness of the “realities of 2054.”

The film, as we know, was a success (by the way, another idea from Minority Report, albeit on a different principle, was tried to be implemented in Britain – this was predicting the names of future killers). But the main thing is that the “future” in this context turned out to be not so distant.

Luminous Room “pumped up its muscles”, gained more advanced “intelligence” and turned into a G-Speak environment. For its further development and promotion to the market, the Oblong Industries company was recently created (Underkoffler became its leading scientist and one of the founders). Now the amazing interface is presented to the public under the slogan “Minority Report in reality”. Advertising, where would we be without it? And the system is truly impressive.

But if you are too lazy to wave your hands for hours on end, you can try contacting Nicole. Literally: “Nicole! Launch IE and please find me Beethoven’s Moonlight Sonata on the Internet.” It will launch and find. And then he will lose, obeying the same voice commands.

The creator of this system, a kind of virtual secretary, or rather desktop personal assistant, is Tom Moir from the School of Engineering and Advanced Technology, Massey University of New Zealand. Yesterday, December 2, Tom showed the Nicole virtual assistant in Auckland, at the 15th international conference on mechatronics and machine vision M2VIP.

Brazilians consider their Nicole the most realistic virtual assistant in the world. The best or not, this character knows how to transform from a secretary into a TV presenter, and can also take off his clothes. But such an “option” is useless in the Moira project. As well as the fact that in fact the Nicole bot is complete, from head to toe, and even capable of putting on different dresses and suits, because in the guise of an assistant, the girl is displayed on the screen in the form of a “talking head”.

Of course, voice control is nothing new in the computer world. Moir’s achievement lies in the fact that he brought this principle, if not to perfection, then, in any case, to a decent level of complexity, and also combined voice control with a virtual character. Here it must be said that the image of Nicole, or rather an entire bot with dozens of facial expressions, was provided to Tom by the Brazilian company Guile 3D Studio, which specializes in “virtual people”.

Nicole’s facial expression can reflect surprise and anger, fun and thoughtfulness, the movement of her lips is consistent with the spoken phrases, she can cry and express doubt in the words of her interlocutor. Perhaps her abilities are even excessive for the role of an on-screen “secretary”. Yes, one of those faces is Tom Moir. Moir taught the Brazilian beauty all sorts of tricks: working with files and folders, launching selected audio and video, searching for data, answering questions like “what time is it?”, reading news aloud, and the like.

But Tom didn’t stop at managing exclusively virtual objects. After all, he believes that Nicole should become an assistant not so much for lazy users as for disabled people who find it difficult or even simply impossible to operate a computer. And if so, the virtual girl should “reach out” to real things. Let’s say, turn on and off a light, a fan, or something like that by voice command. The system created by Moir allows all this.

Well, if you’re tired of communicating with Nicole, try the “thing interface”. The device “Mirror” (Mir:ror) is produced and sold by the French company Violet. For 45 euros, the user gets the opportunity to turn everyday things into interactive objects. The system, which debuted last month, is built on familiar radio frequency identification (RFID) chips.

Typically, these inconspicuous radio markers are used in factories, warehouses and stores to record and sort a large number of objects. But sometimes there are original systems – RFID key fobs, microchips and cards in the role of train tickets, hotel room keys or systems against forgetful surgeons, for example. And still, RFID is somehow not very closely involved in everyday life. But what prospects are opening up!

So, Mir:ror is a desktop reader of special RFID tags called Ztamps (they come included and can be purchased as needed), plus the corresponding software. These microchips can be placed anywhere. After an object equipped with such a chip is brought to the “Mirror”, the computer will execute a set of commands previously “associated” with this object. This principle is very similar to the work of the “subject-based” system of teaching English, only the possibilities of the “Mirror” are much wider. What you program to launch this or that “software” is a matter of taste. The French offer different options: from mugs to sneakers.

The carrier of the RFID chip, which also allows you to run a set of commands from the “Mirror” – the “micro rabbit” Nano:ztag. In appearance, this “beast” replicates the much larger interactive Internet rabbit Nabaztag, which gained a good reputation and fame for Violet several years earlier. He independently accesses the Internet (perceives commands by ear) and has a voice (reads news, exchange rates or stock market reports), movable ears and multi-colored illumination.

For example, a book placed on the “round” of the device will automatically launch the Internet and open a page on the publisher’s website or a video containing additional information, or the writer’s website. The weather forecast for tomorrow can be obtained by holding an umbrella to Mir:ror; If you pick up a souvenir from your last trip, the computer will show the corresponding photo album; your baby’s favorite toy will make the PC read a story; a pack of cigarettes using this device can count the number of cigarettes smoked per day or week; An MP3 player placed on the “Mirror” will open a folder on the computer with a “strategic” reserve of songs, and a mobile phone will open the website of the warranty workshop. Why not? You decide.

“Morning. You place your car keys on this mysterious device. A voice says: “It’s raining today, 14°C. You will get to work in 15 minutes.” The computer screen includes images from web cameras located along the previously planned route, while a voice reads the horoscope for the day,” this is the scenario offered by the manufacturing company.

A computer can carry out a host of instructions programmed by the user simply by “seeing in the mirror” an object associated with that set of actions. Much easier than sitting in front of the screen and sequentially opening all these applications manually. Such is the “little magic”.

It seems to be essentially nothing complicated. We have already seen similar (partly) projects before. In the form of laboratory experiments and “dreams”. But no one, it seems, has yet thought of releasing such a finished product to the market for the masses. Are you still struggling with the keyboard? G-Speak, Nicole, Mir:ror offer an alternative. How convenient it is is another question. But the main thing is when there is a choice, isn’t it?