Neural network dubbing in Ukrainian – ElevenLabs’ new AI feature translates videos into 20 different languages with voice preservation
The system allows you to replace the voice of an actor with an artificial voice that sounds the same as the voice speaking another language.
ElevenLabs is an AI startup that offers a voice cloning service that allows users to generate audio using neural networks based on uploaded samples lasting several minutes. (At the beginning of the year, the company made its technology publicly available, which was immediately used by Internet trolls – fake celebrity voices were spread on social media, saying offensive things, declaring wars, or quoting Hitler.)
Now the service has expanded with a new AI feature that will allow you to translate voice to audio in more than 20 different languages, including Hindi, Portuguese, Spanish, Japanese, Ukrainian, Polish, and Arabic – but legally and with permission. This tool is intended for use by media companies, educators, and individual creators.
ElevenLabs claims that the system will not only be able to translate “spoken content into another language in minutes,” but also create new dialogues in the target language using the actor’s own voice. The system is reportedly even able to maintain the “emotions and intonation” of the underlying dialog and transfer them to the generated translation.
“This will help audiences enjoy any content, regardless of the language they speak,” says ElevenLabs CEO Mati Staniszewski. “And it will mean that content creators can easily and authentically access a much larger audience around the world.”
Earlier, YouTube used AI to dub audio and is currently testing the technology with “hundreds” of authors. The feature is available in English, Spanish, and Portuguese, but the list will be expanded in the future. According to the company, they are “working to make the translated audio tracks sound like the author’s voice, with more expressiveness and synchronization.” The launch of AI dubbing is scheduled for 2024.
