This AI Company Wants to Revolutionize How Dialogue Is Re-Dubbed Into Different Languages
At present, watching a piece of foreign language content that’s been dubbed in English is an exercise in discomfort.
Thanks to the proliferation of local-language film and TV — brought to us mostly by streaming platforms — the experience of viewing a dubbed Danish crime procedural or a Spanish boarding school soap opera are nightmarish trips to the Uncanny Valley. That’s the term widely used for technology that does not suspend our disbelief and, in fact, takes human resemblance and tosses it into a blender of dystopian nightmares.
More from Variety
Enter Flawless, an AI-driven filmmaking studio that wants you to stream buzzy shows and still be able to sleep at night (sans visions of mismatching mouth movements and brutal scene cuts). Founded in 2018 by multi-hyphenate director Scott Mann and Nick Lynes, Flawless’ proprietary tech TrueSync maps over the faces of actors and delivers translations that are the most impressive we’ve seen in the wild west of AI. The company previewed a sizzle reel of its work at this year’s Cannes Film Festival, which Variety can exclusively share here.
Designed to protect artistic copyrights and comply with Hollywood labor guilds, Flawless raises big questions about the value of international content and its breakthrough potential in U.S. markets. Variety can also report that Flawless is officially in the film distribution game. The company has entered a joint venture with XYZ Films and Tea Shop Productions to acquire foreign film rights. The partnership will convert the projects to English language with what Flawless promises to be “perfectly lip-synced visuals” and release them in relevant territories. The first title for release is Victor Dannell’s award-winning sci-fi romp “UFO Sweden,” which will release under the domestic title “Watch the Skies.” A full slate of projects is expected in the coming months*.
Mann spoke with Variety at length about AI fear and loathing in show business, building an ethical code and the brain mechanics of the Uncanny Valley.
How did you get started in this space?
This all began in 2018, My background is directing film, writing and producing. I happened upon what was what turned out to be the first paper that introduced generative AI to the world. It was called “Deep Video Portrait” and came out of the SIGGRAPH Conference, which is basically the Oscars for science. It blew my mind.
I’d always worked in the VFX realm, and it was clear the impact this tech could potentially have on our industry. I reached out to the authors of that paper, which led to the founding of the company with those folks and [my partner] Nick Lynes. Christian Theobalt, Pablo Garrido and Hyeongwoo Kim founded our science department, and have grown it to the leading computer-driven generative science department in the world. That’s our secret sauce.
There’s a lot of fear and anxiety around AI in Hollywood at the moment. What’s interesting about your company is that you’re optimistic.
We founded this thing from a filmmaking point of view. I love art, and essential to our mission is to see this technology as having the ability to make tools. There’s a different view that this aims to eradicate the creatives and steal art. To Napster the hell out of it, to ride roughshod and disrupt for the sake of disruption. But we believe in the expression of human stories, and film is the most advanced form of that. The tech is amplifying that and adding more layers of creativity. There are benefits. Then you have the economy of this whole business, which relies on certain protections and an understanding of how it works.
There’s a big lack of education around AI in this town. How does that impact your business?
This is a global issue. It’s not just Hollywood. Education and understanding are the key There are certain companies — the flawed ones, let’s call it — that would rather not educate because it reveals a kind of underhand player. You need to have an honest conversation about how and where it’s disrupting, and where the positivity is. We’ve prioritized understanding over the past few years with the studios and the labor guilds – including SAG-AFTRA – and said, “Look, this is how this should be used.” The next [AMPTP contract] negotiation is something everyone should be starting to talk about now. There’s this transformational thing entering our industry and we should be prepared.
Good point. Is the industry paying attention to this in a meaningful way?
I would confidently say there’s a sea change in attitudes out there. Before last year’s strikes, there was a cautious route to greenlighting the use of AI in projects. People were having early conversations and seemed to understand them, and it came from a place of trying to improve process. But there is a fear about crossing over. The industry is in such a bad shape in places right now that many don’t want to rush into the wrong solution.
How did Flawless build its ethical code?
It emerged from problems I experienced personally as a filmmaker. I’d always wanted to fix film dubbing. I did a movie where I’d taken such care, a film called “Heist” with Robert De Niro. It was very intimately made and nuanced, and a joy to deliver what we did. I saw the foreign language version that had been dubbed. The script got completely rewritten and the mouth movements were completely different. I realized, “God, no wonder things don’t travel well.”
I got a bug to fix this problem and tried a bunch of VFX like head scanning and other nonsense. It wasn’t practical, and it didn’t cross the Uncanny Valley. This is a long way of saying you can’t do this ethically if you are removing and destroying whole areas that contribute to a film. We’ve learned that ethics and legal rights go hand in hand.
Speaking of the Uncanny Valley, it’s so brutal watching lip-dubbing in its current form. Especially the way streaming has escalated it.
As human beings, we study each other in great detail. If we see something is out of the ordinary, our brains are programmed to go on hyper alert. Throughout the evolution of filmmaking, our brains have gotten fooled and retrained. Your subconscious is doing the work to check for reality. If something doesn’t marry well – particularly sound and vision — you’re immediately putting a huge barrier to any kind of immersion of empathy and believability in what you’re watching. Think of how much expression is conveyed through the face and the gestures and the performance of an actor, without even hearing the sound. There’s so much in that. All those reasons make [present lip dubbing] a problem. But it’s good to have a clear problem to solve.
There’s also a pretty game-changing element here for international films. It could open a new star market and eliminate a lot of U.S. language remakes of celebrated films.
Showing our materials to filmmakers, especially over the past year, they realize the potential from going to a local stage to a global stage. It’s a huge opportunity to get your work out and it’s been invigorating. They are so excited about showing their work in a wider audience, and especially in America.
Has any studio or corporation tried to acquire you outright?
Yes, when we first dropped down here in Hollywood. We had a very short promo reel in 2021 and some tried to broker a price right out of the box. Nick and I agreed that what we want is to serve everyone, not just the interests of one studio. We want to be Switzerland, because it’s very limiting otherwise. But that’s helped us raise money and grow the company. It’s a good problem to have.
You also want to get into the distribution game.
We’ve always looked at ourselves as a studio, to some degree. We want to be future-focused. You need to do the whole thing properly from pre-production, production to release. All these stages are impacted by the technology. My outlook is that we should be getting films to 10x the audience. That, and making amazing films for less cost with these tools. If you look at our first, “UFO Sweden,” it’s obviously been translated to English but also gave the filmmakers freedom to add and change things they couldn’t afford the first time around. You can choose to be like an algorithmic component and try to do one job, but I think the real power comes to making the movies and distributing. The whole shebang. And in terms of post-production, we can do more. Look at Pixar, they get to iterate their films up and to the moment of release.
*Acquisitions for the JV with Flawless will be led Ryan Black, XYZ’s James Emanuel Shapiro and James Harris of Tea Shop.
Best of Variety
Sign up for Variety’s Newsletter. For the latest news, follow us on Facebook, Twitter, and Instagram.
