While the web is full of concepts that combine Pokémon GO with the HoloLens headset and Microsoft does everything it can to attract more users to Windows 10, the excellent research division in Redmond continues developing technologies that could rewrite several rules in virtual reality, augmented reality, and telepresence. This time we're looking at the Fusion4D project, a capture system for complex movements in high definition, with automatic noise filtering, and in real time.
Gesture recognition, motion capture, high-speed and extreme-definition cameras, and of course those with the skill to squeeze the most out of these tools have started a new era in the development of movies, video games, series, and applications. A solid example of these advances is Hellblade, a title we explored a bit more closely last April, and whose behind-the-scenes shows several technical details that are truly impressive. The best part is that the hardware does not come alone in this adventure. Software is increasingly flexible, precise, and intelligent, which sooner or later helps achieve higher-quality productions at lower cost. Some technologies are still trapped in the lab, but thanks to divisions like Microsoft Research, we can see what's coming. Today, it's Fusion4D's turn.
The official description for Fusion4D is something like "high-performance capture of challenging scenes in real time," but that introduction doesn't do justice to what we see in the video. Basically, we're dealing with an algorithm capable of performing coherent, dynamic, high-quality reconstructions in real time. These are very complex processes, especially when considering "non-rigid captures" with a high random profile, such as the movement of a skirt on a dress. In the demo shown in the video, the 3D model was generated from eight depth cameras, on which RGB frames, depth data, and segmentation are combined, assisting in the texturing process. The algorithm's learning ability results in astonishing noise filtering.
The fact that you don't have to establish prior configuration parameters, and the possibility (albeit still remote) of abandoning spandex suits (read lycra or spandex), face dots, and bright lights, makes Fusion4D a technology we need to watch step by step. Interesting fact: Several of the researchers who worked on the project left Microsoft to form a company called PerceptiveIO. Will we see a commercial solution?
