PIXELES TACTILES

Throughout the history of computers we've been striving to shorten the gap between us and digital information, the gap between our physical world and the world in the screen where our imagination can go wild. And this gap has become shorter, shorter, and even shorter,and now this gap is shortened down to less than a millimeter, the thickness of a touchscreen glass, and the power of computing has become accessible to everyone. But I wondered, what if there could be no boundary at all? I started to imagine what this would look like. First, I created this tool which penetrates into the digital space, so when you press it hard on the screen, it transfers its physical body into pixels. Designers can materialize their ideas directly in 3D, and surgeons can practice on virtual organsunderneath the screen. So with this tool, this boundary has been broken. But our two hands still remain outside the screen. How can you reach inside and interactwith the digital information using the full dexterity of our hands? At Microsoft Applied Sciences, along with my mentor Cati Boulanger, I redesigned the computer and turned a little space above the keyboard into a digital workspace. By combining a transparent display and depth cameras for sensing your fingers and face, now you can lift up your hands from the keyboard and reach inside this 3D space and grab pixels with your bare hands. (Applause) Because windows and files have a position in the real space, selecting them is as easy as grabbing a book off your shelf. Then you can flip through this book while highlighting the lines, words on the virtual touch pad below each floating window. Architects can stretch or rotate the models with their two hands directly. So in these examples, we are reaching into the digital world. But how about reversing its role and having the digital information reach us instead? I'm sure many of us have had the experience of buying and returning items online. But now you don't have to worry about it. What I got here is an online augmented fitting room. This is a view that you get from head-mounted or see-through display when the system understands the geometry of your body. Taking this idea further, I started to think, instead of just seeing these pixels in our space,how can we make it physical so that we can touch and feel it? What would such a future look like? At MIT Media Lab, along with my advisor Hiroshi Ishii and my collaborator Rehmi Post, we created this one physical pixel. Well, in this case, this spherical magnet acts like a 3D pixel in our space, which means that both computers and people can move this object to anywhere within this little 3D space. What we did was essentially canceling gravity and controlling the movement by combining magnetic levitation and mechanical actuation and sensing technologies. And by digitally programming the object, we are liberating the object from constraints of time and space, which means that now, human motions can be recorded and played back and left permanently in the physical world. So choreography can be taught physically over distance and Michael Jordan's famous shooting can be replicated over and over as a physical reality. Students can use this as a tool to learn about the complex concepts such as planetary motion, physics, and unlike computer screens or textbooks,this is a real, tangible experience that you can touch and feel, and it's very powerful. And what's more exciting than just turning what's currently in the computer physical is to start imagining how programming the world will alter even our daily physical activities. (Laughter) As you can see, the digital information will not just show us something but it will start directly acting upon us as a part of our physical surroundings without disconnecting ourselves from our world. Today, we started by talking about the boundary, but if we remove this boundary, the only boundary left is our imagination. Thank you. (Applause)

Desde los comienzos de las computadoras nos hemos venido esforzando por reducir la separacin entre nosotros y la informacin digital, la separacin entre nuestro mundo material y el mundo de la pantalla en el que la imaginacin se puede desbocar. Esta separacin se ha ido reduciendo, cada vez ms y ms, a tal punto que hoy en da es de menos de 1 mm, el espesor del vidrio de una pantalla tctil, y el poder de la informtica se ha vuelto accesible a todos. Pero me pregunto: y si no hubiera barreras? Empec a imaginar cmo sera. Primero cre esta herramienta que se adentra en el espacio digital, de modo que cuando la oprimes fuerte contra la pantalla transfiere su cuerpo fsico a pxeles. Los diseadores pueden materializar sus ideas directamente en 3D, y los cirujanos pueden practicar con rganos virtuales debajo de la pantalla. As, con esta herramienta se rompen las barreras. Pero las dos manos permanecen todava por fuera de la pantalla. Cmo llegar al interior e interactuar con la informacin digital usando toda la destreza de las manos? En la divisin de Ciencias Aplicadas de Microsoft, junto con mi mentora Cati Boulanger, redise la computadora y transform un pequeo espacio sobre el teclado en un rea digital de trabajo. Combinando un visor transparente con cmaras de profundidad para detectar los dedos y la cara, ahora puedes levantar las manos del teclado, llegar al interior del espacio 3D, y agarrar pxeles directamente con las manos. (Aplausos) Como las ventanas y los archivos tienen una ubicacin en el espacio real, seleccionarlos es tan fcil como agarrar un libro de un estante. Se puede hojear el libro y resaltar lneas o palabras con el sensor tctil virtual que hay abajo de cada pantalla flotante. Los arquitectos pueden estirar o rotar sus maquetas directamente con las manos. En estos ejemplos,nosotros ingresamos en el mundo digital. Y si invertimos los papeles y hacemos que la informacin digital venga hacia nosotros?Seguramente, muchos de nosotros habremos comprado y devuelto cosas por Internet.Ahora eso no tiene que preocuparnos. Lo que tengo ac es un probador virtual por Internet.Esta es la visin que se obtiene desde un dispositivo montado en la cabeza o translcidocuando el sistema comprende la geometra de tu cuerpo.

Si llevamos esta idea ms lejos, pens, en vez de solo ver pxeles en el espacio, cmo podemos hacer que sean fsicos de modo que podamos tocarlos y sentirlos? Cmo sera un futuro as? En el Media Lab del MIT con mi tutor Hiroshi Ishii y mi colaborador Rehmi Post, creamos este nico pxel fsico. Este imn esfrico se comporta como un pxel en 3D en el espacio, lo que implica que tanto la computadora como los usuarios pueden mover el objeto a cualquier lugar dentro de este pequeo espacio tridimensional. Simplificando, lo que hicimos fue anular la gravedad y controlar el movimiento mediante una combinacin delevitacin magntica, accionamiento mecnico y detectores. Y al programar digitalmente el objeto, lo liberamos de las restricciones del tiempo y el espacio, lo que quiere decir que los movimientos humanos pueden grabarse y volver a reproducirse y quedan para siempre en el mundo fsico. Se pueden ensear coreografas fsicamente y a distancia y la famosa toma de Michael Jordan se puede reproducir como una realidad fsica las veces que queramos.Los estudiantes pueden usarlo como herramienta para entender conceptos complicadoscomo el movimiento de los planetas, la fsica, y, a diferencia de los monitores o los libros de texto, esta es una experiencia real y palpable que puedes tocar y sentir. Es muy poderosa. Pero lo que es ms fascinante que cambiar la parte fsica de la computadora es imaginar cmo programar el mundo va a cambiar nuestras actividades fsicas cotidianas. (Risas) Como ven, la informacin digital no solo nos mostrar algo sino que comenzar a actuar directamente sobre nosotros como parte del mundo fsico que nos rodea sin que tengamos que desconectarnos de nuestro mundo. Empezamos la charla de hoy hablando de una barrera, pero si suprimimos esa barrera, el nico lmite que queda es nuestra imaginacin.

QUADCOPTERS
So what does it mean for a machine to be athletic? We will demonstrate the concept of machine athleticism and the research to achieve it with the help of these flying machines called quadrocopters, or quads, for short. Quads have been around for a long time. The reason that they're so popular these days is because they're mechanically simple. By controlling the speeds of these four propellers,these machines can roll, pitch, yaw, and accelerate along their common orientation. On board are also a battery, a computer, various sensors and wireless radios. Quads are extremely agile, but this agility comes at a cost. They are inherently unstable, and they need some form of automatic feedback control in order to be able to fly. So, how did it just do that? Cameras on the ceiling and a laptop serve as an indoor global positioning system. It's used to locate objects in the space that have these reflective markers on them. This data is then sent to another laptop that is running estimation and control algorithms, which in turn sends commands to the quad, which is also running estimation and control algorithms. The bulk of our research is algorithms. It's the magic that brings these machines to life. So how does one design the algorithms that create a machine athlete? We use something broadly called model-based design. We first capture the physics with a mathematical modelof how the machines behave. We then use a branch of mathematics called control theory to analyze these models and also to synthesize algorithms for controlling them. For example, that's how we can make the quad hover. We first captured the dynamics with a set of differential equations. We then manipulate these equations with the help of control theory to create algorithms that stabilize the quad. Let me demonstrate the strength of this approach. Suppose that we want this quad to not only hover but to also balance this pole. With a little bit of practice, it's pretty straightforward for a human being to do this, although we do have the advantage of having two feet on the ground and the use of our very versatile hands. It becomes a little bit more difficult when I only have one foot on the ground and when I don't use my hands. Notice how this pole has a reflective marker on top, which means that it can be located in the space. (Applause) You can notice that this quad is making fine adjustments to keep the pole balanced. How did we design the algorithms to do this? We added the mathematical model of the pole to that of the quad. Once we have a model of the combined quad-pole system, we can use control theory to create algorithms for controlling it. Here, you see that it's stable, and even if I give it little nudges, it goes back to the nice, balanced position. We can also augment the model to include where we want the quad to be in space. Using this pointer, made out of reflective markers, I can point to where I want the quad to be in space a fixed distance away from me. The key to these acrobatic maneuvers is algorithms,designed with the help of mathematical models and control theory. Let's tell the quad to come back here and let the pole

drop, and I will next demonstrate the importance of understanding physical models and the workings of the physical world. Notice how the quad lost altitude when I put this glass of water on it. Unlike the balancing pole, I did not includethe mathematical model of the glass in the system. In fact, the system doesn't even know that the glass of water is there. Like before, I could use the pointer to tell the quad where I want it to be in space. (Applause) Okay, you should be asking yourself, why doesn't the water fall out of the glass? Two facts: The first is that gravity acts on all objects in the same way. The second is that the propellers are all pointing in the same direction of the glass, pointing up. You put these two things together, the net result is that all side forces on the glass are small and are mainly dominated by aerodynamic effects, which as these speeds are negligible. And that's why you don't need to model the glass. It naturally doesn't spill no matter what the quad does. (Applause) The lesson here is that some high-performance tasks are easier than others, and that understanding the physics of the problem tells you which ones are easy and which ones are hard. In this instance, carrying a glass of water is easy. Balancing a pole is hard. We've all heard stories of athletes performing feats while physically injured. Can a machine also perform with extreme physical damage? Conventional wisdom says that you need at least four fixed motor propeller pairs in order to fly, because there are four degrees of freedom to control: roll, pitch, yaw and acceleration. Hexacopters and octocopters, with six and eight propellers, can provide redundancy, but quadrocopters are much more popularbecause they have the minimum number of fixed motor propeller pairs: four. Or do they? If we analyze the mathematical model of this machine with only two working propellers, we discover that there's an unconventional way to fly it. We relinquish control of yaw, but roll, pitch and acceleration can still be controlled with algorithms that exploit this new configuration. Mathematical models tell us exactly when and why this is possible. In this instance, this knowledge allows us to design novel machine architectures or to design clever algorithms that gracefully handle damage, just like human athletes do, instead of building machines with redundancy. We can't help but hold our breath when we watch a diver somersaulting into the water, or when a vaulter is twisting in the air, the ground fast approaching. Will the diver be able to pull off a rip entry? Will the vaulter stick the landing? Suppose we want this quad here to perform a triple flip and finish off at the exact same spot that it started. This maneuver is going to happen so quickly that we can't use position feedback to correct the motion during execution. There simply isn't enough time. Instead, what the quad can do is perform the maneuver blindly, observe how it finishes the maneuver, and then use that information to modify its behavior so that the next flip is better. Similar to the diver and the vaulter, it is only through repeated practice that the maneuver can be learned and executed to the highest standard. (Applause) Striking a moving ball is a necessary skill in many sports. How do we make a machine dowhat an athlete does seemingly without effort? (Applause) This quad has a racket strapped onto its head with a sweet spot roughly the size of an apple, so not too large. The following calculations are made every 20 milliseconds, or 50 times per second. We first figure out where the ball is going. We then next calculate how the quad should hit the ball so that it flies to where it was thrown from. Third, a trajectory is planned that carries the quad from its current state to the impact point with the ball. Fourth, we only execute 20 milliseconds' worth of that strategy. Twenty milliseconds later, the whole process is repeated until the quad strikes the ball. (Applause) Machines can not only perform dynamic maneuvers on their own, they can do it collectively.These three quads are cooperatively carrying a sky net. (Applause) They perform an extremely dynamic and collective maneuver to launch the ball back to me.Notice that, at full extension, these quads are vertical. (Applause) In fact, when fully extended, this is roughly five times greater than what a bungee jumper feels at the end of their launch. The algorithms to do this are very similar to what the single quad used to hit the ball back to me. Mathematical models are used to continuously re-plan a cooperative strategy 50 times per second.

Everything we have seen so far has been about the machines and their capabilities. What happens when we couple this machine athleticism with that of a human being? What I have in front of me is a commercial gesture sensor mainly used in gaming. It can recognize what my various body parts are doing in real time. Similar to the pointer that I used earlier, we can use this as inputs to the system. We now have a natural way of interacting with the raw athleticism of these quads with my gestures. (Applause) Interaction doesn't have to be virtual. It can be physical. Take this quad, for example. It's trying to stay at a fixed point in space. If I try to move it out of the way, it fights me, and moves back to where it wants to be. We can change this behavior, however. We can use mathematical models to estimate the force that I'm applying to the quad. Once we know this force, we can also change the laws of physics, as far as the quad is concerned, of course. Here the quad is behaving as if it were in a viscous fluid. We now have an intimate way of interacting with a machine. I will use this new capability to position this camera-carrying quad to the appropriate location for filming the remainder of this demonstration. So we can physically interact with these quads and we can change the laws of physics.Let's have a little bit of fun with this. For what you will see next, these quads will initially behave as if they were on Pluto. As time goes on, gravity will be increased until we're all back on planet Earth, but I assure you we won't get there. Okay, here goes. (Laughter) (Laughter) (Applause) Whew! You're all thinking now, these guys are having way too much fun, and you're probably also asking yourself, why exactly are they building machine athletes? Some conjecture that the role of play in the animal kingdom is to hone skills and develop capabilities. Others think that it has more of a social role, that it's used to bind the group.Similarly, we use the analogy of sports and athleticism to create new algorithms for machines to push them to their limits. What impact will the speed of machines have on our way of life? Like all our past creations and innovations, they may be used to improve the human condition or they may be misused and abused. This is not a technical choice we are faced with; it's a social one. Let's make the right choice, the choice that brings out the best in the future of machines, just like athleticism in sports can bring out the best in us. Let me introduce you to the wizards behind the green curtain. They're the current members of the Flying Machine Arena research team. (Applause) Federico Augugliaro, Dario Brescianini , Markus Hehn, Sergei Lupashin, Mark Muller and Robin Ritz. Look out for them. They're destined for great things. Thank you.