Setting up a “sounds like me AND writes like me” voice system

Cloning yourself well takes two independent layers, and it’s worth being clear about which is which:

Layer Question it answers What it is Where it lives
Chippify What would I say, and how would I phrase it? A script-style profile distilled from your own talking D:\GitHub\Chippify
chipp-speak What do I sound like? A Qwen3-TTS voice-timbre clone ~/.claude/skills/chipp-speak

They’re separable and you can build either alone — but the magic is the pipeline: Chippify writes the words → chipp-speak speaks them. A perfect voice clone reading robotic AI-marketing copy still doesn’t sound like you; matching both the wording and the timbre is what sells it.

This guide covers Part A — the voice clone (chipp-speak) first, then Part B — the script-style profile (Chippify), then a closing prompt that has Claude build both for you.


Part A — The voice clone (Qwen3-TTS, isolated venv)

This builds a Claude Code skill that generates speech in a cloned voice using Qwen3-TTS, running in an isolated Python venv so it never fights with ComfyUI’s dependencies. It’s the chipp-speak design: a canonical Python synth + a bash wrapper that splits, synthesizes, concatenates, and normalizes in one pass.

The core idea: ComfyUI ships transformers 5.x, but the Qwen3-TTS backend needs transformers >=4.57,<5. Downgrading globally breaks other ComfyUI nodes. So we build a separate venv that imports qwen_tts directly — no ComfyUI, no HTTP — while still reusing the models ComfyUI already downloaded. Nothing gets re-downloaded; the isolation is only at the Python level.


What you’ll end up with

~/.claude/skills/<your-skill>/
├── skill.md                       # skill definition Claude reads
└── scripts/
    ├── <voice>_tts_synth.py       # canonical synth (imports qwen_tts directly)
    └── generate-speech.sh         # wrapper: split → synth → concat → normalize

Plus, living outside the skill (so they survive skill backups and are shared):

  • An isolated venv, e.g. D:\qwen-tts-venv
  • A clean voice reference clip + its transcript
  • The Qwen3-TTS model weights (reused from ComfyUI’s models dir)

Prerequisites

  • Windows with an NVIDIA GPU (this guide assumes CUDA; CPU works but is slow).
  • ffmpeg / ffprobe on PATH (used for concat + loudness normalization).
  • Git Bash (or another bash) — the wrapper is a .sh script and uses cygpath to hand Windows paths to ffmpeg.
  • Python 3.12 available to create a venv.
  • The Qwen3-TTS model weights already downloaded (ComfyUI users usually have these under …/ComfyUI/models/qwen-tts/). If you don’t, download Qwen3-TTS-12Hz-1.7B-Base (and optionally the 0.6B) into a folder you’ll point the synth at.

Step 1 — Record a clean voice reference

The clone quality is almost entirely determined by this one clip.

  1. Record 15–30 seconds of clear speech, no background noise, no music, natural pace. Read normal prose, not a word list.
  2. Convert it to 16 kHz mono WAV:
    ffmpeg -i raw_recording.wav -ac 1 -ar 16000 myvoice_ref16k.wav
    
  3. Write the exact transcript of what you said into a matching .txt:
    myvoice_ref16k.txt   ← the precise words spoken in the wav
    
  4. Put both somewhere stable that any skill can read. (chipp-speak keeps them in ComfyUI’s input/ dir so multiple skills share them without depending on each other.) Note the full absolute paths — you’ll hard-code them as defaults in Step 3.

Step 2 — Build the isolated venv

Create a venv that is only for TTS, and install a transformers version the Qwen3-TTS backend is happy with.

# Pick a stable location OUTSIDE the skill folder
py -3.12 -m venv D:\qwen-tts-venv
D:\qwen-tts-venv\Scripts\python.exe -m pip install --upgrade pip

# Install CUDA torch (match your CUDA — cu128 shown here), the TTS backend,
# and audio I/O deps. Pin transformers below 5.
D:\qwen-tts-venv\Scripts\python.exe -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
D:\qwen-tts-venv\Scripts\python.exe -m pip install "transformers>=4.57,<5" qwen-tts librosa soundfile numpy

Why a separate venv and not just pip install into ComfyUI? Because ComfyUI’s transformers 5.x is load-bearing for QwenVL/Wan and other nodes. Keep them apart. This venv never touches ComfyUI’s python.

Sanity-check the import:

D:\qwen-tts-venv\Scripts\python.exe -c "from qwen_tts import Qwen3TTSModel; print('ok')"

Step 3 — Write the canonical synth

Create scripts/<voice>_tts_synth.py. It takes text + out.wav, loads the model in the isolated venv, clones from your reference, and writes a WAV. Hard-code your model dir and reference paths as defaults so callers stay simple.

Key points (mirroring qwen_tts_synth.py):

  • MODELS_DIR points at the already-downloaded weights (reuse ComfyUI’s).
  • DEFAULT_REF / DEFAULT_REF_TEXT point at your clip + transcript from Step 1.
  • Load the reference as 16 kHz mono via librosa, pass (wav, sr) to model.generate_voice_clone(...).
  • Print Output: <path> on success so the wrapper can parse it.
  • Use device_map="cuda", dtype=torch.bfloat16, attn_implementation="sdpa".
  • A fixed --seed (e.g. 42) makes renders reproducible.

You can copy scripts/qwen_tts_synth.py from chipp-speak and just change the three path constants (MODELS_DIR, DEFAULT_REF, DEFAULT_REF_TEXT).


Step 4 — Write the wrapper

Create scripts/generate-speech.sh. This is where the quality tuning lives — it’s not just a thin shell. Copy chipp-speak’s wrapper and adjust the python path and synth filename. What it does, and why:

  1. Accepts text or a .txt path — if [ -f "$INPUT" ] decides.
  2. Dashes → commas before synth (—, –, -). Bare hyphens make the TTS say “A-O”.
  3. Paragraph split, then sentence-batch anything over WORD_LIMIT=55 words. Qwen3-TTS hallucinates on long single renders — keep each segment short.
  4. Synth each segment to its own WAV via the isolated venv python.
  5. Concatenate, then loudnorm ONCE over the whole clip (loudnorm=I=-16:TP=-1.5:LRA=7) → MP3 192k. Normalizing per-segment causes level pumping between paragraphs; one pass keeps it consistent.
  6. Uses cygpath -m so Windows ffmpeg can read the concat list and WAV paths.
  7. Fails loud — exits with a clear error if the venv/synth is missing. Optional edge provider is a disclosed free fallback, never a silent one.

Step 5 — Write skill.md

This is the file Claude reads to know when and how to run the skill. Minimum frontmatter + a short body:

---
name: <your-skill>
description: Generate speech in <Name>'s cloned voice using Qwen3-TTS in an
  isolated venv (NOT ComfyUI). Use when the user wants speech that sounds like
  <Name>. Accepts text or a .txt path, paragraph-splits long input, normalizes
  once, outputs MP3.
---

# <your-skill> — <Name> voice clone TTS

Generates speech in <Name>'s cloned voice with Qwen3-TTS via the isolated venv.

## Usage
bash scripts/generate-speech.sh "<text or path/to/script.txt>" "C:/path/out.mp3"

## Notes
- Long docs are fine — paragraph splitting handles them; keep blank-line breaks
  so pauses land naturally.
- GPU is serialized; the model loads per segment.
- Fails loud if the venv or synth is missing — don't silently fall back.

Keep the description specific and trigger-focused — that’s what Claude matches on to decide whether to invoke the skill.


Step 6 — Test

bash ~/.claude/skills/<your-skill>/scripts/generate-speech.sh \
  "Hi, this is a test of my cloned voice." \
  "C:/Users/<you>/Downloads/voice_test.mp3"

You should see segment-by-segment progress, then a final Speech saved: … with duration + size. Play it:

powershell.exe -c "Start-Process 'C:/Users/<you>/Downloads/voice_test.mp3'"

Gotchas & tips

  • Reference clip is everything. If the clone sounds off, re-record a cleaner, longer (but ≤30s), more natural reference before touching anything else.
  • Don’t share one venv with ComfyUI. Pin transformers <5 in the TTS venv only.
  • Keep the venv, models, and reference OUTSIDE the skill folder. Skills get backed up/synced; you don’t want gigabytes of weights or a venv in that repo. Reference them by absolute path.
  • One normalization pass, at the end. Never normalize per segment.
  • 55-word ceiling is a real Qwen3-TTS limit — don’t raise it hoping for fewer segments; you’ll get hallucinations instead.
  • Reuse, don’t re-download model weights if you already have them.

Part B — The script-style profile (Chippify)

The voice clone gives you timbre. Chippify gives you wording — a reusable profile that makes Claude write scripts that sound like you talking, not like generic AI copy. It’s built by mining your own past talking and distilling the patterns. (Chipp’s profile lives in the repo D:\GitHub\Chippify; the v0.2 profile itself is D:\GitHub\Chippify\CHIPPIFY_SCRIPT_PROFILE.md.)

Why this matters: Chipp’s corpus showed his natural cadence is ~11 words per spoken chunk, heavy on “so / now / okay / right? / kind of / I’m going to go ahead and”, with casual openings (“Hey guys, Chipp here…”) and a casual close (“Thanks for watching, hope to see you online. Bye.”). Without a profile, an LLM writes 30-word academic sentences that the TTS then dutifully reads in your voice — and it lands in the uncanny valley. The profile fixes the words.

How Chippify was built (the workflow to copy)

Step 1 — Collect a corpus of YOU talking

Gather 15–20 recordings where you speak naturally in the voice you want to reproduce (tutorials, walkthroughs, talks). For Chipp these were YouTube videos, pulled with yt-dlp to audio/<id>__<title>.mp3 plus an .info.json of metadata per video, tracked in metadata/selected_manifest.json.

Curate for authenticity. Chippify v0.2 was regenerated after removing 5 videos whose transcripts looked AI-generated / over-polished — they were dragging the profile toward marketing-speak. Keep only recordings that sound like the real, unscripted you. Garbage in, uncanny out.

Step 2 — Transcribe with fillers preserved

Transcribe each clip with faster-whisper (large-v3, CUDA, word timestamps ON). The crucial setting: VAD filter OFF, so quiet fillers and pauses (“uh”, “so”, “kind of”) are not dropped — those disfluencies ARE the style signal. Chipp’s script is D:\GitHub\Chippify\scripts\transcribe_chippify.py; it writes one <id>.json (segments + word timings) and <id>.txt per video. Result: ~114k words across ~9,090 timestamped segments.

# example shape (run in a venv that has faster-whisper + CUDA torch)
python D:\GitHub\Chippify\scripts\transcribe_chippify.py --model large-v3 --device cuda

Step 3 — Analyze the cadence statistically

Count what actually characterizes your speech, don’t guess. Chipp’s analysis (D:\GitHub\Chippify\metadata\chippify_style_analysis_notes.md) tallied:

  • Segment length — median 11 words, mean 12.5 → “write in short spoken beats.”
  • Filler/cadence frequencies — so ×2136, let’s ×1091, just ×883, now ×747, right ×694, okay ×415, “kind of” ×255, uh ×166 (vs. um only ×15 → prefer “uh”).
  • Sentence starts — “I’m going to…”, “and then”, “all right so…”, “you can see”.
  • Signature openings — pulled verbatim from the corpus.

This is what turns vague vibes into concrete, enforceable rules.

Step 4 — Distill into a profile document

Fold those findings into a single Markdown profile an LLM can follow. Chipp’s CHIPPIFY_SCRIPT_PROFILE.md includes: a voice summary, what the voice is not (corporate explainer, AI promo, hype influencer), the cadence facts, a high-frequency marker list with usage rules, signature phrases, opening patterns, a 7-part tutorial structure, rewrite rules, generation rules, an output format, and — most reusable — a drop-in Claude Project instruction and a compact prompt card at the bottom.

Provenance note: Chipp’s profile itself was authored in a claude.ai web Project (not a Claude Code session) — the tell is the CLAUDE_PROJECT_INSTRUCTIONS_DRAFT.txt file. You can build yours either way; the corpus → transcribe → analyze → distill workflow is the same.

Step 5 — Wire it into your skills

Have your video/voice skills read the profile before drafting any script. Chipp’s hole-in-one skill was edited to read D:\GitHub\Chippify\CHIPPIFY_* before writing a tutorial script, so every generated voiceover is Chippified first, then synthesized in his cloned voice. That edit is the seam where Part B feeds Part A.

The complete pipeline

  your past talking (videos/recordings)
        │  yt-dlp
        ▼
  audio corpus  ──► faster-whisper (VAD off, word ts) ──► transcripts
        │                                                      │
        │                                  statistical analysis │
        ▼                                                      ▼
  16kHz ref clip + transcript                   CHIPPIFY_SCRIPT_PROFILE.md
        │                                                      │
   ┌────┴─────────── Part A ───────────┐         ┌──────────── Part B ──────────┐
   │  Qwen3-TTS voice clone (timbre)   │         │  script-style profile (words)│
   └───────────────┬───────────────────┘         └──────────────┬───────────────┘
                   │                                             │
                   │   new script ◄── Claude writes it using ────┘
                   │                  the profile (Chippified)
                   ▼
        generate-speech.sh  ──►  MP3 that sounds like you, saying it like you

Prompt: have Claude build this for your voice

This builds both layers — the script-style profile (writes like you) and the voice clone (sounds like you). Paste it into Claude Code and fill in the bracketed bits first. If you only want one layer, delete the other part.

I want to clone my own voice the way the existing setup clones Chipp's. There are
TWO layers and I want both:
  A) a Qwen3-TTS VOICE CLONE skill (sounds like me), modeled on the `chipp-speak`
     skill at C:\Users\chipp\.claude\skills\chipp-speak\
  B) a SCRIPT-STYLE PROFILE (writes like me), modeled on the Chippify repo at
     D:\GitHub\Chippify\

First, READ these so you copy the architecture exactly:
  - C:\Users\chipp\.claude\skills\chipp-speak\SETUP.md   (the full guide)
  - C:\Users\chipp\.claude\skills\chipp-speak\skill.md
  - C:\Users\chipp\.claude\skills\chipp-speak\scripts\qwen_tts_synth.py
  - C:\Users\chipp\.claude\skills\chipp-speak\scripts\generate-speech.sh
  - D:\GitHub\Chippify\CHIPPIFY_SCRIPT_PROFILE.md
  - D:\GitHub\Chippify\scripts\transcribe_chippify.py
  - D:\GitHub\Chippify\metadata\chippify_style_analysis_notes.md

=== Part A — voice clone ===
- New skill name: [my-voice]            (e.g. "jamie-speak")
- My voice reference WAV (16kHz mono):  [C:\path\to\myvoice_ref16k.wav]
- The exact transcript of that WAV:     [C:\path\to\myvoice_ref16k.txt]
  (or: here is the transcript text — create the .txt for me: "[...]" )
- Isolated venv python:                 [D:\qwen-tts-venv\Scripts\python.exe]
  (or: "I don't have the venv yet — give me the exact commands to build it,
   pinning transformers >=4.57,<5, then continue.")
- Qwen3-TTS model weights dir:          [D:\...\ComfyUI\models\qwen-tts]
  (or: "I don't have the weights — tell me what to download and where.")

Requirements for Part A:
1. Create ~/.claude/skills/[my-voice]/ with skill.md + scripts/ (a renamed copy
   of the synth and the wrapper), pointing the path constants at MY paths above.
2. Keep ALL the quality behavior from chipp-speak: dashes→commas, 55-word
   paragraph/sentence splitting, single post-concat loudnorm
   (I=-16:TP=-1.5:LRA=7), MP3 192k output, cygpath for ffmpeg, and fail-loud
   errors (no silent fallback).
3. Make the skill.md description trigger on "speech that sounds like [Name]".
4. If my reference clip isn't already 16kHz mono, give me the ffmpeg command to
   convert it.

=== Part B — script-style profile ===
- My corpus of me-talking lives at:     [folder of video/audio files, or a list
                                          of YouTube URLs]
  (or: "I don't have a corpus yet — tell me exactly how to collect ~15-20 clips
   with yt-dlp and where to put them.")
- Build a profile repo at:              [D:\GitHub\MyVoice]   (or another path)

Requirements for Part B:
1. Transcribe the corpus the Chippify way: faster-whisper large-v3, word
   timestamps ON, VAD OFF (so my fillers survive). Reuse/adapt transcribe_chippify.py.
2. Statistically analyze cadence: segment-length median/mean, filler frequencies,
   common sentence starts, and pull representative openings verbatim.
3. Distill it all into MY_SCRIPT_PROFILE.md, including a drop-in instruction block
   and a compact prompt card at the bottom, exactly like Chippify's.
4. Flag and EXCLUDE any clips that read as AI-generated/over-polished — I want my
   real unscripted voice, not marketing copy.

=== Wire them together ===
After both exist, show me how to have my video/voice skills READ the profile to
write a script first, THEN synthesize it with the [my-voice] clone — so the
output both reads like me and sounds like me. Finish with a short end-to-end test:
write a ~30s script from a topic I'll give you, Chippify it with MY profile, speak
it with MY clone, and play the result.

Walk me through anything you need from me, then build it.