AI-Generated Voiceovers Used in State Influence Operations
Generative AI has been widely known for its misuse in various areas, from creating fake academic papers to copying artists’ works. However, a recent report by Massachusetts-based threat intelligence company Recorded Future has shed light on its involvement in state influence operations.
The report details a Russian-tied campaign called “Operation Undercut,” aimed at undermining Europe’s support for Ukraine. This campaign heavily relied on AI-generated voiceovers in fake or misleading news videos targeting European audiences. These videos spread misinformation about Ukrainian politicians and questioned the effectiveness of military aid to Ukraine.
Recorded Future’s researchers found that the video creators likely used commercial AI voice generation products, including technology from ElevenLabs, to make their content appear more authentic. The use of AI allowed the misleading clips to be quickly released in multiple European languages, such as English, French, German, Polish, and Turkish.
Although ElevenLabs did not respond to requests for comment, its AI Speech Classifier was used to confirm the use of its technology in the creation of these fake videos. The influence campaign’s orchestrators inadvertently revealed the use of AI-generated voiceovers by releasing videos with real human voiceovers that had a discernible Russian accent, while the AI-generated voiceovers spoke in multiple European languages without any foreign-sounding accents.
According to Recorded Future, the Social Design Agency, a Russia-based organization sanctioned by the U.S. government, was behind this activity. The impact of the campaign on public opinion in Europe was deemed minimal by Recorded Future.
This is not the first time ElevenLabs’ products have been linked to misuse. The company was previously involved in a robocall impersonating President Joe Biden during a primary election, prompting new safety features to be implemented to prevent such incidents in the future.
ElevenLabs has strict policies against unauthorized impersonation and deceptive practices, utilizing automated and human moderation tools to enforce them. The company has experienced rapid growth since its inception in 2022, attracting investments from notable firms like Andreessen Horowitz and former Github CEO Nat Friedman.
