Voice cloning

In short

Voice cloning is generating speech that resembles a particular person's voice from a recording of them. A model learns a speaker representation, then combines it with new words, language or delivery instructions.

Also called: AI voice cloning, Voice synthesis, Zero-shot voice cloning
Topic: Images, video, voice and music
Level: Beginner
First described: Arık and colleagues published Neural Voice Cloning with a Few Samples at NeurIPS in 2018, explicitly studying the task under the name voice cloning.

A voice-cloning system usually separates what is said from who is saying it. A reference recording is converted into speaker information, while text or another speech signal supplies the words and timing. A speech generator then produces an audio waveform conditioned on both. Some systems adapt a model to a speaker; others infer a speaker embedding directly from a short sample¹.

Arık and colleagues' 2018 paper studied neural voice cloning from only a few audio samples. It compared speaker adaptation, which fine-tunes a multi-speaker model, with speaker encoding, which infers a new speaker representation for a model that has already learned many voices¹. Later systems such as YourTTS extended zero-shot cloning to multilingual, multi-speaker speech and reported fine-tuning with less than a minute of speech².

Voice cloning is not the same as playing back a recording, and it does not prove that the real person spoke the generated words. Similarity, pronunciation, emotion and language can vary with the reference audio and model. It also creates consent, impersonation and fraud risks, so access controls and disclosure matter.

It appears in dubbing, accessibility, audiobooks, games and voice assistants. ElevenLabs, for example, requires verification for its Professional Voice Clone and says users may create one only of their own voice; another person can share a verified clone privately³.

Example

A narrator records a short sample, then uses a voice-cloning service to make the same voice read an approved translation of a museum audio guide in another language.

Why it matters

It can make spoken content easier to produce and personalise. The same ability can make a fake phone call or statement sound like a real person, so permission and labelling are central safeguards.

Questions people ask

How much audio is needed to clone a voice?

There is no single amount. The 2018 Neural Voice Cloning paper studied cloning from a few samples, while YourTTS reported fine-tuning with less than one minute. Quality depends on the model, recording and intended use¹².

Is voice cloning the same as text-to-speech?

Text-to-speech turns written text into spoken audio. Voice cloning is a form of text-to-speech or voice conversion that adds the target speaker's vocal identity, usually from a reference recording¹.

Can I clone someone else's voice?

Policies differ by service and local law. ElevenLabs says its Professional Voice Clone can only be made from the user's own voice, even with another person's consent; a verified owner can share the clone privately³.

Related terms

Tools that use it

Sources

Back to the glossary