On Cloning a VoiceCloning a Voice · Tools Set the Ceiling, Craft Sets the Floor

This began with something I had misremembered. I thought Qwen had released a model that could read books in your voice from a dozen or so seconds of recorded audio. It turned out I had mixed things up: Qwen handles conversation and multimodal tasks; voice cloning comes from another branch of Alibaba's Tongyi family, called CosyVoice. And it does not even need a dozen or so seconds. In the official example, 3 and a half seconds is enough.

I had no clean recordings of my own, so I borrowed two people's voices for testing: one goes by LZK, the other by SCD. Each recorded a dozen or so seconds. Every “reference clip” and “cloned output” below was made using their recordings, not my voice. I wanted to make that clear before anyone listened and got the wrong idea.

CosyVoice, Not Qwen

A whole crop of open-source voice-cloning models that run locally has appeared over the past two years. GPT-SoVITS is the most popular in the Chinese-speaking community, with more tutorials on Bilibili than anyone could watch. Fish Speech and F5-TTS cover multiple languages. For the very best quality, there are ElevenLabs and ByteDance's Seed-TTS, but both are closed-source and cost money.

What I wanted was specific: take a dozen or so seconds of someone's voice, have it read a text several thousand Chinese characters long, and save that voice to use again. CosyVoice 2/3 is the easiest fit for this work; it was designed for long texts and saved voice profiles. GPT-SoVITS might sound more like the person in a single sentence, but the voice drifts over several thousand characters. It also needs a round of fine-tuning, which is a nuisance.

I also spent a while wondering which version to use. My instinct was, “3 is newer than 2, and 1.5B is bigger than 0.5B, so go with the biggest.” A little checking put that to rest: across the CosyVoice series, only 0.5B has been open-sourced. The supposed 1.5B is just a model size used for comparison in the paper; its weights were never released. Besides, TTS does not follow the LLM logic of more parameters meaning more intelligence. Its limits lie in the data and architecture, and 0.5B is already very capable at Chinese voice cloning. I settled on Fun-CosyVoice3-0.5B.

The Five Snags That Took the Time

When setting up an environment, the main procedure is never where the time goes. It all goes into the parts nobody wrote down. I got stuck five times:

• The brew cask for installing miniforge had long been retired. It threw a pile of Ruby stack traces; switching to the official script got me through;
• That old version of openai-whisper still uses pkg_resources in setup.py, which newer setuptools no longer provides. The catch was that downgrading setuptools did not fix the error: pip isolates builds by default and reinstalls the newer version in a temporary environment. In the end, I had to install that package separately with --no-build-isolation;
• The permission system blocked pip from installing packages. I tried asking the AI to write itself an allow rule, and got a firm refusal: “You cannot change your own permission configuration,” regardless of who approves it. That was the right restriction. A tool should not be able to give itself a back door;
• I was sure ModelScope had CosyVoice3. It only had 2;
• I switched to HuggingFace and ran into another snag with the mirror: hf-mirror now sends a 308 redirect back to the original site. After a lot of fuss, I found that connecting directly to the official site worked.

The nine GB of weights finally downloaded, and the smoke test passed. Loading the model from a cold start took 73 seconds; synthesizing 5 seconds of audio took 8 seconds. It worked.

The Machine First Speaks in Someone Else's Voice

I started with LZK's recording. First, listen to the original: him reading aloud for sixteen seconds:

Then I had the model use that voice to read a science-fiction short story it had never seen, “Echo Star.” Below are the first 45 seconds of the result. The full story is 1445 Chinese characters and runs 5 minutes 55 seconds. It took 23 minutes to generate on the Mac:

One detail was rather counterintuitive. When transcribing LZK's recording, whisper misheard the Chinese rendering of Wozniak, “Woziniyake,” as “Wosinike,” and Jobs, “Qiaobusi,” as “Qiaofusi.” At first I thought this would ruin the clone, but it was fine: the model matches sounds, not written characters. Misspelling a proper name does not matter as long as the pronunciation lines up. Knowing this can save a lot of needless worrying.

One Voice, Eighteen Dialects

This is the most startling part of CosyVoice3: with the same reference clip, it can switch dialects or languages while keeping the voice's timbre. It claims to support 18 Chinese dialects and 9 languages. All six clips below use the same LZK voice to say the same sentence, “The weather is lovely today. Let's go for a walk in the park,” in six different dialects:

Notice the length of each clip. Mandarin takes 5 seconds; the Tianjin dialect takes only 3.8 seconds. The same sentence has six different lengths. That shows it is not simply applying a filter to the voice, but actually reading the sentence again with each dialect's inflection and pace. This was more convincing than I had expected.

Clean Reference Audio Matters More Than Length

The sample from the second person, SCD, is more interesting because it did not work at first. At the end of the recording, SCD casually added an exclamation that was not in Mandarin. Using the whole recording as a reference pulled the cloned voice's timbre and tone off course.

The fix was to trim it using whisper's word-level timestamps. The Mandarin section ends at 12.76 seconds, and the exclamation begins at 12.92 seconds. I cut at 12.85 seconds, leaving 0.09 seconds of breathing room. That kept the end of the last syllable intact without letting the unwanted sound in. Here is SCD's reference clip after trimming:

I used this clean reference clip to read the fable “Covering His Ears to Steal a Bell”:

So with reference audio, clean matters more than long. A steady, even 10 seconds is better than 15 seconds with noise, dialect speech, and a sudden shift in emotion.

Is a Mac Enough?

Mine is an M5 Pro. In testing, the real-time factor ranged from 2.66 to 3.88, meaning it took 2 to 4 seconds to synthesize 1 second of audio. Usable, though hardly fast. That is where those 23 minutes for “Echo Star” went.

I also worked out what an older RTX 2060 (6GB) graphics card could do. It can run the model, with a real-time factor of 0.3 to 0.5, close to real time and five to eight times faster than the M5. The gap is less about raw computing power than about how much more mature PyTorch's CUDA ecosystem is than Apple's MPS. The conclusion is fairly plain: a Mac will do for the occasional audiobook. For real-time use or several hours of generation a day, even a secondhand NVIDIA card is better value than a Mac.

The Full Setup Procedure

Here are both the Mac and Windows procedures, with fixes for all the snags I hit today worked in. Copy them and you should be able to get it running.

Mac (Apple Silicon)

# 1. System audio tools
brew install ffmpeg sox

# 2. Install miniforge (the cask is retired; use the official script)
curl -fsSL -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh"
bash Miniforge3-MacOSX-arm64.sh -b -p $HOME/miniforge3

# 3. Create an environment; pynini must come from conda-forge (it will not build with pip)
source ~/miniforge3/etc/profile.d/conda.sh
conda create -n cosyvoice -y python=3.10 pynini=2.1.6 -c conda-forge
conda activate cosyvoice

# 4. Clone the code (including submodules)
git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git
cd CosyVoice

# 5. Install dependencies (work around the whisper issue here)
pip install "setuptools<70" wheel
pip install numpy==1.26.4 numba tqdm more-itertools tiktoken
pip install --no-build-isolation openai-whisper==20231117
pip install -r requirements.txt

# 6. Download the model directly from the official site; do not use hf-mirror
python -c "from huggingface_hub import snapshot_download; snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B', max_workers=4)"

Windows (with an NVIDIA graphics card, such as a 2060)

# 1. Install Miniforge3 for Windows (download the Windows x86_64 installer from the official site and click Next through the steps)
#    Then run the following commands in the “Miniforge Prompt” terminal

# 2. Create an environment; use conda-forge for pynini here too (pip cannot build it on Windows either)
conda create -n cosyvoice -y python=3.10 pynini=2.1.6 -c conda-forge
conda activate cosyvoice

# 3. Clone the code
git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git
cd CosyVoice

# 4. Install the CUDA version of PyTorch first (this corresponds to the cu121 line in requirements)
pip install torch==2.3.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121

# 5. The same whisper issue, with the same workaround
pip install "setuptools<70" wheel
pip install numpy==1.26.4 numba tqdm more-itertools tiktoken
pip install --no-build-isolation openai-whisper==20231117

# 6. Install the remaining dependencies and the GPU version of onnxruntime
pip install -r requirements.txt
pip uninstall -y onnxruntime
pip install onnxruntime-gpu==1.18.0

# 7. Download the model
python -c "from huggingface_hub import snapshot_download; snapshot_download('FunAudioLLM/Fun-CosyVoice3-0.5B-2512', local_dir='pretrained_models/Fun-CosyVoice3-0.5B', max_workers=4)"

A few other details for Windows: the graphics driver must be at least version 525 for CUDA 12.1 support; 6GB of VRAM is the minimum. For long texts, break the sentences into shorter pieces and turn off the ttsfrd text frontend, or it is easy to run out of VRAM. Avoid WSL; the disk I/O is painfully slow. Run it natively on Windows.

Looking back, it began with something I had misremembered. After correcting my choice of model, working through five snags, and getting both people's voices to work, I ended up with a local pipeline that can read any long Chinese text in anyone's voice with a few lines of configuration changed. The model itself is certainly capable. But what really determines the quality of the result is what the README leaves out: how clean the reference audio is, whether the transcript matches, and where the sentences are split.

Translation log: translated from the Chinese original by Codex on 2026-11-03, 17:43–17:46 (US Eastern). Main thread GPT-6 Astra (gpt-6-astra), reasoning effort ultra; it spawned 1 sub-thread: 1 on GPT-6 Astra (effort ultra). Together they read 630,622 tokens (594,432 of them from cache) and wrote 10,358 (2,535 of them reasoning).