Cinema and Deepfake: zooming in on lip syncing with foreign language dubbing

Deepfake is a technique based on artificial intelligence that allows audio or video files to be superimposed on other already existing files to reproduce a person's voice or change their face and therefore make them say say or do things that these individuals have never done. This technology is often used to create hoaxes or to convey fake news. However, it could be used wisely in the cinema for dubbing actors in several languages.

AI software to work around lip dubbing and jointing issues

Usually, deepfake is used to alter a person's facial features, gestures, or voice in an image or video, with the aim of making them say something they have never said or done in reality. If this practice is actually associated with hoaxes and internet memes, it could eventually be used for cinema to allow for more realistic and easy to achieve dubbing.

The British company Flawless has announced the development of software, called TrueSync, which can analyze the facial patterns of actors and modify them so that they can integrate into the vocalization of another language. To be more precise: one of the recurring problems of dubbing consists in making so that the lyrics in the dubbed language can stick with the movement of the lips. A difficulty that the British firm's AI experts want to alleviate.

I added a video to a @YouTube playlist https://t.co/fiygt0y2OC How to Paint a Guitar With Metal Flake

— tony rodriguez Tue May 16 19:26:25 +0000 2017

This is explained by Scott Mann, one of the directors who decided to exploit the technology of Flawless:

Deepfake, a useful practice in certain circumstances, subject to ethical questions

TrueSync enables realistic synchronization by adapting the mouth and lip movement of the original actors to the articulation that can be found in several foreign languages. A few weeks ago, Flawless released a video where you could see some results of using the tool to dub movies with famous actors like Robert de Niro, Tom Hanks, Jack Nicholson, and Tom Cruise.

However, the use of software like this presents several ethical issues in particular. Some professionals have noted that if the software falls into the wrong hands, it could be exploited to commit crimes. In addition, it must also be assumed that the actors must accept that their lips (and therefore part of their body) are modified to facilitate dubbing in other languages.

Several questions that will undoubtedly push back the future and massive use of deepfake techniques in the cinema.