# Setting up a "sounds like me AND writes like me" voice system

Cloning yourself well takes **two independent layers**, and it's worth being
clear about which is which:

| Layer | Question it answers | What it is | Where it lives |
|-------|--------------------|-----------|----------------|
| **Chippify** | *What would I say, and how would I phrase it?* | A **script-style profile** distilled from your own talking | `D:\GitHub\Chippify` |
| **chipp-speak** | *What do I sound like?* | A **Qwen3-TTS voice-timbre clone** | `~/.claude/skills/chipp-speak` |

They're separable and you can build either alone — but the magic is the pipeline:
**Chippify writes the words → chipp-speak speaks them.** A perfect voice clone
reading robotic AI-marketing copy still doesn't sound like you; matching *both*
the wording and the timbre is what sells it.

This guide covers **Part A — the voice clone** (chipp-speak) first, then
**Part B — the script-style profile** (Chippify), then a closing prompt that has
Claude build both for *you*.

---

# Part A — The voice clone (Qwen3-TTS, isolated venv)

This builds a Claude Code skill that generates speech in a **cloned voice** using
Qwen3-TTS, running in an **isolated Python venv** so it never fights with
ComfyUI's dependencies. It's the `chipp-speak` design: a canonical Python synth +
a bash wrapper that splits, synthesizes, concatenates, and normalizes in one pass.

> **The core idea:** ComfyUI ships `transformers 5.x`, but the Qwen3-TTS backend
> needs `transformers >=4.57,<5`. Downgrading globally breaks other ComfyUI
> nodes. So we build a *separate* venv that imports `qwen_tts` directly — no
> ComfyUI, no HTTP — while still **reusing the models ComfyUI already
> downloaded**. Nothing gets re-downloaded; the isolation is only at the Python
> level.

---

## What you'll end up with

```
~/.claude/skills/<your-skill>/
├── skill.md                       # skill definition Claude reads
└── scripts/
    ├── <voice>_tts_synth.py       # canonical synth (imports qwen_tts directly)
    └── generate-speech.sh         # wrapper: split → synth → concat → normalize
```

Plus, living **outside** the skill (so they survive skill backups and are shared):

- An isolated venv, e.g. `D:\qwen-tts-venv`
- A clean voice reference clip + its transcript
- The Qwen3-TTS model weights (reused from ComfyUI's models dir)

---

## Prerequisites

- **Windows** with an NVIDIA GPU (this guide assumes CUDA; CPU works but is slow).
- **ffmpeg / ffprobe** on PATH (used for concat + loudness normalization).
- **Git Bash** (or another bash) — the wrapper is a `.sh` script and uses
  `cygpath` to hand Windows paths to ffmpeg.
- **Python 3.12** available to create a venv.
- The **Qwen3-TTS model weights** already downloaded (ComfyUI users usually have
  these under `…/ComfyUI/models/qwen-tts/`). If you don't, download
  `Qwen3-TTS-12Hz-1.7B-Base` (and optionally the `0.6B`) into a folder you'll
  point the synth at.

---

## Step 1 — Record a clean voice reference

The clone quality is almost entirely determined by this one clip.

1. Record **15–30 seconds** of clear speech, no background noise, no music,
   natural pace. Read normal prose, not a word list.
2. Convert it to **16 kHz mono WAV**:
   ```bash
   ffmpeg -i raw_recording.wav -ac 1 -ar 16000 myvoice_ref16k.wav
   ```
3. Write the **exact transcript** of what you said into a matching `.txt`:
   ```
   myvoice_ref16k.txt   ← the precise words spoken in the wav
   ```
4. Put both somewhere stable that any skill can read. (chipp-speak keeps them in
   ComfyUI's `input/` dir so multiple skills share them without depending on
   each other.) Note the **full absolute paths** — you'll hard-code them as
   defaults in Step 3.

---

## Step 2 — Build the isolated venv

Create a venv that is *only* for TTS, and install a `transformers` version the
Qwen3-TTS backend is happy with.

```powershell
# Pick a stable location OUTSIDE the skill folder
py -3.12 -m venv D:\qwen-tts-venv
D:\qwen-tts-venv\Scripts\python.exe -m pip install --upgrade pip

# Install CUDA torch (match your CUDA — cu128 shown here), the TTS backend,
# and audio I/O deps. Pin transformers below 5.
D:\qwen-tts-venv\Scripts\python.exe -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128
D:\qwen-tts-venv\Scripts\python.exe -m pip install "transformers>=4.57,<5" qwen-tts librosa soundfile numpy
```

> **Why a separate venv and not just `pip install` into ComfyUI?** Because
> ComfyUI's `transformers 5.x` is load-bearing for QwenVL/Wan and other nodes.
> Keep them apart. This venv never touches ComfyUI's python.

Sanity-check the import:
```powershell
D:\qwen-tts-venv\Scripts\python.exe -c "from qwen_tts import Qwen3TTSModel; print('ok')"
```

---

## Step 3 — Write the canonical synth

Create `scripts/<voice>_tts_synth.py`. It takes `text` + `out.wav`, loads the
model **in the isolated venv**, clones from your reference, and writes a WAV.
Hard-code your model dir and reference paths as defaults so callers stay simple.

Key points (mirroring `qwen_tts_synth.py`):

- `MODELS_DIR` points at the **already-downloaded** weights (reuse ComfyUI's).
- `DEFAULT_REF` / `DEFAULT_REF_TEXT` point at **your** clip + transcript from Step 1.
- Load the reference as 16 kHz mono via `librosa`, pass `(wav, sr)` to
  `model.generate_voice_clone(...)`.
- Print `Output: <path>` on success so the wrapper can parse it.
- Use `device_map="cuda"`, `dtype=torch.bfloat16`, `attn_implementation="sdpa"`.
- A fixed `--seed` (e.g. 42) makes renders reproducible.

You can copy `scripts/qwen_tts_synth.py` from chipp-speak and just change the
three path constants (`MODELS_DIR`, `DEFAULT_REF`, `DEFAULT_REF_TEXT`).

---

## Step 4 — Write the wrapper

Create `scripts/generate-speech.sh`. This is where the **quality tuning** lives —
it's not just a thin shell. Copy chipp-speak's wrapper and adjust the python path
and synth filename. What it does, and *why*:

1. **Accepts text or a `.txt` path** — `if [ -f "$INPUT" ]` decides.
2. **Dashes → commas** before synth (`—`, `–`, ` - `). Bare hyphens make the TTS
   say "A-O".
3. **Paragraph split, then sentence-batch** anything over `WORD_LIMIT=55` words.
   Qwen3-TTS **hallucinates on long single renders** — keep each segment short.
4. **Synth each segment** to its own WAV via the isolated venv python.
5. **Concatenate, then `loudnorm` ONCE** over the whole clip
   (`loudnorm=I=-16:TP=-1.5:LRA=7`) → MP3 192k. Normalizing per-segment causes
   **level pumping between paragraphs**; one pass keeps it consistent.
6. Uses `cygpath -m` so Windows ffmpeg can read the concat list and WAV paths.
7. **Fails loud** — exits with a clear error if the venv/synth is missing.
   Optional `edge` provider is a *disclosed* free fallback, never a silent one.

---

## Step 5 — Write `skill.md`

This is the file Claude reads to know when and how to run the skill. Minimum
frontmatter + a short body:

```markdown
---
name: <your-skill>
description: Generate speech in <Name>'s cloned voice using Qwen3-TTS in an
  isolated venv (NOT ComfyUI). Use when the user wants speech that sounds like
  <Name>. Accepts text or a .txt path, paragraph-splits long input, normalizes
  once, outputs MP3.
---

# <your-skill> — <Name> voice clone TTS

Generates speech in <Name>'s cloned voice with Qwen3-TTS via the isolated venv.

## Usage
bash scripts/generate-speech.sh "<text or path/to/script.txt>" "C:/path/out.mp3"

## Notes
- Long docs are fine — paragraph splitting handles them; keep blank-line breaks
  so pauses land naturally.
- GPU is serialized; the model loads per segment.
- Fails loud if the venv or synth is missing — don't silently fall back.
```

Keep the `description` specific and trigger-focused — that's what Claude matches
on to decide whether to invoke the skill.

---

## Step 6 — Test

```bash
bash ~/.claude/skills/<your-skill>/scripts/generate-speech.sh \
  "Hi, this is a test of my cloned voice." \
  "C:/Users/<you>/Downloads/voice_test.mp3"
```

You should see segment-by-segment progress, then a final
`Speech saved: …` with duration + size. Play it:
```powershell
powershell.exe -c "Start-Process 'C:/Users/<you>/Downloads/voice_test.mp3'"
```

---

## Gotchas & tips

- **Reference clip is everything.** If the clone sounds off, re-record a cleaner,
  longer (but ≤30s), more natural reference before touching anything else.
- **Don't share one venv with ComfyUI.** Pin `transformers <5` in the TTS venv only.
- **Keep the venv, models, and reference OUTSIDE the skill folder.** Skills get
  backed up/synced; you don't want gigabytes of weights or a venv in that repo.
  Reference them by absolute path.
- **One normalization pass, at the end.** Never normalize per segment.
- **55-word ceiling** is a real Qwen3-TTS limit — don't raise it hoping for
  fewer segments; you'll get hallucinations instead.
- **Reuse, don't re-download** model weights if you already have them.

---

# Part B — The script-style profile (Chippify)

The voice clone gives you *timbre*. Chippify gives you *wording* — a reusable
profile that makes Claude write scripts that sound like **you talking**, not like
generic AI copy. It's built by mining your own past talking and distilling the
patterns. (Chipp's profile lives in the repo `D:\GitHub\Chippify`; the v0.2
profile itself is `D:\GitHub\Chippify\CHIPPIFY_SCRIPT_PROFILE.md`.)

> **Why this matters:** Chipp's corpus showed his natural cadence is ~11 words
> per spoken chunk, heavy on "so / now / okay / right? / kind of / I'm going to
> go ahead and", with casual openings ("Hey guys, Chipp here…") and a casual
> close ("Thanks for watching, hope to see you online. Bye."). Without a profile,
> an LLM writes 30-word academic sentences that the TTS then dutifully reads in
> your voice — and it lands in the uncanny valley. The profile fixes the *words*.

## How Chippify was built (the workflow to copy)

### Step 1 — Collect a corpus of YOU talking

Gather 15–20 recordings where you speak naturally in the voice you want to
reproduce (tutorials, walkthroughs, talks). For Chipp these were YouTube videos,
pulled with `yt-dlp` to `audio/<id>__<title>.mp3` plus an `.info.json` of
metadata per video, tracked in `metadata/selected_manifest.json`.

> **Curate for authenticity.** Chippify v0.2 was *regenerated* after removing 5
> videos whose transcripts looked AI-generated / over-polished — they were
> dragging the profile toward marketing-speak. Keep only recordings that sound
> like the real, unscripted you. Garbage in, uncanny out.

### Step 2 — Transcribe with fillers preserved

Transcribe each clip with **faster-whisper** (`large-v3`, CUDA, word timestamps
ON). The crucial setting: **VAD filter OFF**, so quiet fillers and pauses
("uh", "so", "kind of") are *not* dropped — those disfluencies ARE the style
signal. Chipp's script is `D:\GitHub\Chippify\scripts\transcribe_chippify.py`;
it writes one `<id>.json` (segments + word timings) and `<id>.txt` per video.
Result: ~114k words across ~9,090 timestamped segments.

```powershell
# example shape (run in a venv that has faster-whisper + CUDA torch)
python D:\GitHub\Chippify\scripts\transcribe_chippify.py --model large-v3 --device cuda
```

### Step 3 — Analyze the cadence statistically

Count what actually characterizes your speech, don't guess. Chipp's analysis
(`D:\GitHub\Chippify\metadata\chippify_style_analysis_notes.md`) tallied:

- **Segment length** — median 11 words, mean 12.5 → "write in short spoken beats."
- **Filler/cadence frequencies** — so ×2136, let's ×1091, just ×883, now ×747,
  right ×694, okay ×415, "kind of" ×255, uh ×166 (vs. um only ×15 → prefer "uh").
- **Sentence starts** — "I'm going to…", "and then", "all right so…", "you can see".
- **Signature openings** — pulled verbatim from the corpus.

This is what turns vague vibes into concrete, enforceable rules.

### Step 4 — Distill into a profile document

Fold those findings into a single Markdown profile an LLM can follow. Chipp's
`CHIPPIFY_SCRIPT_PROFILE.md` includes: a voice summary, what the voice is **not**
(corporate explainer, AI promo, hype influencer), the cadence facts, a
high-frequency marker list *with usage rules*, signature phrases, opening
patterns, a 7-part tutorial structure, rewrite rules, generation rules, an output
format, and — most reusable — a **drop-in Claude Project instruction** and a
**compact prompt card** at the bottom.

> **Provenance note:** Chipp's profile itself was authored in a **claude.ai web
> Project** (not a Claude Code session) — the tell is the
> `CLAUDE_PROJECT_INSTRUCTIONS_DRAFT.txt` file. You can build yours either way;
> the corpus → transcribe → analyze → distill workflow is the same.

### Step 5 — Wire it into your skills

Have your video/voice skills **read the profile before drafting any script**.
Chipp's `hole-in-one` skill was edited to read `D:\GitHub\Chippify\CHIPPIFY_*`
before writing a tutorial script, so every generated voiceover is Chippified
first, *then* synthesized in his cloned voice. That edit is the seam where Part B
feeds Part A.

## The complete pipeline

```
  your past talking (videos/recordings)
        │  yt-dlp
        ▼
  audio corpus  ──► faster-whisper (VAD off, word ts) ──► transcripts
        │                                                      │
        │                                  statistical analysis │
        ▼                                                      ▼
  16kHz ref clip + transcript                   CHIPPIFY_SCRIPT_PROFILE.md
        │                                                      │
   ┌────┴─────────── Part A ───────────┐         ┌──────────── Part B ──────────┐
   │  Qwen3-TTS voice clone (timbre)   │         │  script-style profile (words)│
   └───────────────┬───────────────────┘         └──────────────┬───────────────┘
                   │                                             │
                   │   new script ◄── Claude writes it using ────┘
                   │                  the profile (Chippified)
                   ▼
        generate-speech.sh  ──►  MP3 that sounds like you, saying it like you
```

---

## Prompt: have Claude build this for *your* voice

This builds **both layers** — the script-style profile (writes like you) and the
voice clone (sounds like you). Paste it into Claude Code and fill in the
bracketed bits first. If you only want one layer, delete the other part.

```
I want to clone my own voice the way the existing setup clones Chipp's. There are
TWO layers and I want both:
  A) a Qwen3-TTS VOICE CLONE skill (sounds like me), modeled on the `chipp-speak`
     skill at C:\Users\chipp\.claude\skills\chipp-speak\
  B) a SCRIPT-STYLE PROFILE (writes like me), modeled on the Chippify repo at
     D:\GitHub\Chippify\

First, READ these so you copy the architecture exactly:
  - C:\Users\chipp\.claude\skills\chipp-speak\SETUP.md   (the full guide)
  - C:\Users\chipp\.claude\skills\chipp-speak\skill.md
  - C:\Users\chipp\.claude\skills\chipp-speak\scripts\qwen_tts_synth.py
  - C:\Users\chipp\.claude\skills\chipp-speak\scripts\generate-speech.sh
  - D:\GitHub\Chippify\CHIPPIFY_SCRIPT_PROFILE.md
  - D:\GitHub\Chippify\scripts\transcribe_chippify.py
  - D:\GitHub\Chippify\metadata\chippify_style_analysis_notes.md

=== Part A — voice clone ===
- New skill name: [my-voice]            (e.g. "jamie-speak")
- My voice reference WAV (16kHz mono):  [C:\path\to\myvoice_ref16k.wav]
- The exact transcript of that WAV:     [C:\path\to\myvoice_ref16k.txt]
  (or: here is the transcript text — create the .txt for me: "[...]" )
- Isolated venv python:                 [D:\qwen-tts-venv\Scripts\python.exe]
  (or: "I don't have the venv yet — give me the exact commands to build it,
   pinning transformers >=4.57,<5, then continue.")
- Qwen3-TTS model weights dir:          [D:\...\ComfyUI\models\qwen-tts]
  (or: "I don't have the weights — tell me what to download and where.")

Requirements for Part A:
1. Create ~/.claude/skills/[my-voice]/ with skill.md + scripts/ (a renamed copy
   of the synth and the wrapper), pointing the path constants at MY paths above.
2. Keep ALL the quality behavior from chipp-speak: dashes→commas, 55-word
   paragraph/sentence splitting, single post-concat loudnorm
   (I=-16:TP=-1.5:LRA=7), MP3 192k output, cygpath for ffmpeg, and fail-loud
   errors (no silent fallback).
3. Make the skill.md description trigger on "speech that sounds like [Name]".
4. If my reference clip isn't already 16kHz mono, give me the ffmpeg command to
   convert it.

=== Part B — script-style profile ===
- My corpus of me-talking lives at:     [folder of video/audio files, or a list
                                          of YouTube URLs]
  (or: "I don't have a corpus yet — tell me exactly how to collect ~15-20 clips
   with yt-dlp and where to put them.")
- Build a profile repo at:              [D:\GitHub\MyVoice]   (or another path)

Requirements for Part B:
1. Transcribe the corpus the Chippify way: faster-whisper large-v3, word
   timestamps ON, VAD OFF (so my fillers survive). Reuse/adapt transcribe_chippify.py.
2. Statistically analyze cadence: segment-length median/mean, filler frequencies,
   common sentence starts, and pull representative openings verbatim.
3. Distill it all into MY_SCRIPT_PROFILE.md, including a drop-in instruction block
   and a compact prompt card at the bottom, exactly like Chippify's.
4. Flag and EXCLUDE any clips that read as AI-generated/over-polished — I want my
   real unscripted voice, not marketing copy.

=== Wire them together ===
After both exist, show me how to have my video/voice skills READ the profile to
write a script first, THEN synthesize it with the [my-voice] clone — so the
output both reads like me and sounds like me. Finish with a short end-to-end test:
write a ~30s script from a topic I'll give you, Chippify it with MY profile, speak
it with MY clone, and play the result.

Walk me through anything you need from me, then build it.
```
