Snapchat calls them by the official name Lenses, but every social network must keep up with its users, who simply nicknamed them filters. These filters apply different effects and elements in real time, which allows them to cross the border of augmented reality without major hiccups, but how exactly do they work? In short, a lot of facial recognition, a battalion of algorithms, and the processing power of modern smartphones make this possible.
A flower crown that isn't there, face swaps, ghost and demon eyes, dog noses... little digital nonsense rendered in real time, with the potential to completely change the experience on this social network, and make more than one wonder: How do they do that?
How Snapchat filters work
According to this video published by Vox, the technology was assimilated by Snapchat through the acquisition of a Ukrainian company called Looksery. Multiple sources report that Snapchat paid $150 million for Looksery, the largest acquisition ever recorded in the Ukrainian technology market to date. In general terms, this is a clear example of object recognition (after all, our face is an object to the system), which assigns a specific color value to each pixel, and from there calculates and interprets the different areas of contrast in a face.
The 'Viola-Jones detection framework'
The video mentions the Viola-Jones detection framework, the first of its kind, whose development began in 2001. Thanks to long training sessions and multiple analyses, the algorithm can recognize very specific aspects of a face, since the front of the nose tends to be lighter than its sides, the 'band' of the eyes is darker, and the forehead is brighter.
However, that is not enough. Viola-Jones has the ability to determine if there is a face in the image or not, but the patents point to an active shape model, essentially a statistical model of a face that was manually trained, highlighting the edges of facial features in thousands and thousands of public samples. The algorithm takes a standard model as reference, places it over the user's face, modifies its scale and angle, applies dynamic adjustments based on the data presented by the pixels, and thus obtains a three-dimensional mask that reacts according to the information obtained in each frame transmitted by the camera.
This turns our face into a series of anchor points for effects, deformations, and animation triggers (for example, a long tongue when we open our mouth). The most demanding function among the filters is face swapping, and due to the enormous amount of information that must be processed, the results are not always optimal.
Basically, Snapchat filters rely on a concept with almost twenty years of development, which was finally implemented in real time, thanks to multi-core processors, accelerated video modules, and entire gigabytes of RAM in our smartphones.
