In an era overrun by fake news, the concept of deepfake is here to stay. In some cases, they did nothing more than celebrate certain famous figures, while in others they tried to undress them. What's next on the list? For now, creating music videos using multiple short clips processed with lip sync. This is possible thanks to the Wav2Lip system, which, in addition to being open source, already has an interactive demo online.
At first, many were intimidated by the technical details behind deepfake, but we all knew it was a bomb about to explode. Bans, censorship, deleted forums, removed videos… nothing stopped enthusiasts. Projects multiplied, and some presented very interesting results. For example, deepfakes came to the rescue of The Mummy Returns, enabling the improvement of its horrible CGI, and another user managed to erase the famous “damn mustache” of Henry Cavill.
And today we arrive at the gates of Wav2Lip, a project by Rudrabha Mukhopadhyay that serves as the basis for the study “A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild”, published on August 23. In relaxed terms, the goal is to adapt an audio piece to any available video with an advanced lip sync process, without sacrificing precision.
As if that were not enough, this works with any identity, voice, and language, including synthetic voices and computer-generated faces. The team behind Wav2Lip has shared several videos on YouTube, but it's the users who manage to elevate these tools to another level. That said, YouTube user “ontyj” admits he got carried away during his tests with Wav2Lip and ended up creating this marvel:
Interested? If so, we have good news for you. Wav2Lip has an interactive demo. All you need to do is upload a video (20 seconds maximum) and an audio sample (same 20-second limit). With a bit of luck, the results will be satisfactory, but if you want to sync longer videos or with higher resolution, you'll have to download and use the code yourself.
Official site: Click here
Interactive demo: Click here
