Whenever an actor interacts with a scenario that's impossible to recreate in real life, we know they're performing in front of a green screen, leaving animation experts to handle the rest. But flip that situation around, and virtual characters can't do much in real environments. A new MIT project aims to change that by studying the vibrations recorded on objects using just a few minutes of video, resulting in surprisingly well-crafted interactive scenarios.
Visual technology applied to film and television is becoming increasingly advanced and affordable. Motion capture, 3D reproduction, digital actors, scenes rendered in real time... the list goes on. A movie that costs 100 million dollars in special effects today will probably require less than half of that in a few years, and I admit I'm being conservative. What else can be done? Where can we improve? The researchers at MIT have decided to explore the interaction of virtual characters with real environments captured in images or video. The problem is that these formats don't provide additional information. For example, filming a static skeleton only gives us 2D data, but if we could analyze its vibrations, the story would change completely.
Making Video Interactive Through Vibrations
Before virtual characters can interact more closely with video, the first step is to make the video itself interactive, and that's where vibrations come in. With their help, the new algorithms developed at MIT can extract additional physical properties about an object and predict how it will behave under external forces. The results published in the presentation video are spectacular. Just five seconds of video are enough for a statuette to be moved and bent. Vibrations are recorded with light taps, but even the effect of wind on leaves or a plant provides plenty of data for creating interactive videos.
At this point, our virtual characters step in. No matter how crude they are, the simple act of hitting an object, walking on it, or jumping on it generates movements very close to what we'd expect in reality. The project's researchers admit the simulations aren't perfect, but they hope that in the future this technology can go beyond its current artistic role and become a tool for monitoring the integrity of structures like bridges or buildings, not to mention its potential in augmented reality.
