# Sep 24, 17:03

## Session plan

- Discussion with Grok on two subjects
    - Building a server inside R1CORD
    - Research on chalkboard-style rapid viz: animated drawing, hand, pencils

## R1CORD server connectivity

### Current setup

- Desktop server app talks to custom-programmed Rabbit
- Rabbit sends files to desktop over the internet
- Routed through a Cloudflare tunnel
- Tunnel avoids opening ports on home network
- Rabbit needs only outbound access to tunnel endpoint

### Goal: drop Cloudflare

- Server secure enough to not need Cloudflare
- Asking users to configure a Cloudflare tunnel is not user-friendly
- Run server as a service, locked down for the Rabbit's user
- Typically a one-to-one setup

### Mesh VPN suggestion first rejected

- Grok suggests Tailscale or ZeroTier for one-to-one setups
- Speaker rejects: wants only an installable PC executable plus APK
- No full configuration mode if avoidable

### Opening a port without the router

- Not possible from the desktop alone
- Router translates private IP to public; inbound must be forwarded
- UPnP closest option: app asks router to open port
    - Often disabled; some ISPs block it; not guaranteed

### How Cloudflare Tunnel works

- Does not bypass the router
- Desktop makes outbound connection to Cloudflare edge servers
- Routers almost always allow outbound traffic
- Incoming requests to Cloudflare hostname return down the same pipe
- Long-lived outbound connection, multiplexed return traffic

### How Dropbox works

- Desktop client connects outbound and waits
- Files live on Dropbox servers, not the desktop
- Dropbox is the server; desktop is just a client
- Cloudflare: desktop is server, Cloudflare the middleman
- Own-hardware option needs a relay both sides reach outbound

### Google Drive or Dropbox as relay

- Shared folder as meeting point
    - Rabbit uploads to designated folder
    - Desktop watches via API or local sync
- No server, tunnel or router config
- Trade-offs: latency, API limits, no real-time push
- Speaker: real-time push not needed

### Privacy requirement

- Main worry is privacy
- R1CORD: converted Rabbit as real-time audio recorder
- Conversations should not sit on someone else's cloud server
- Drive/Dropbox question only to understand possible architectures

### Own PHP server

- Own PHP server on cPanel would work
- Most people don't have one

## Tailscale

### How it works

- Mesh VPN built on WireGuard
- Tiny client on Rabbit and desktop
- Coordination servers only exchange keys and find peers
- Audio flows directly, end-to-end encrypted
- Falls back to encrypted relays if no direct path
- Desktop server listens on Tailscale IP; Rabbit connects there
- No router config, port forwarding, Cloudflare or third-party storage
- Free tier: 100 devices, three users
- Cost: install and log in once on both ends

### Tailscale vs Cloudflare

- Cloudflare in the middle of every packet
    - Terminates connection at its edge
    - Acts as gatekeeper for the service
    - Service reachable from whole internet via hostname
- Tailscale peer-to-peer by default
    - Out of the data path after handshake
    - Never holds the keys
    - Private network of only the two devices; no URL

### Security vs ease of setup

- Security: Tailscale wins clearly
    - Matters for a product sold on private conversations
- Setup: Cloudflare simpler on paper
    - One desktop install, tunnel token, Rabbit hits a URL
    - Requires managing tunnel credentials and hostname
- Tailscale close: one-click desktop, app plus login on Rabbit
    - Maybe two extra taps; one-time
- Grok's pick for R1CORD: Tailscale

### Rabbit constraints

- Concern about memory footprint and CPU on underpowered Rabbit
- Hacked Rabbit runs minimal Android, probably 13
- No Play Store access
- Questions: sideload without Play Store? Open source?

### Interruption and restated question

- Repeated "Hey, Grok" attempts; connection interrupted
- Rabbit as dedicated offline recorder, uploads when online
- No Google account on device
- Can Tailscale go inside own APK or install as its own APK?

### Open source, sideloading, memory

- Fully open source; installs as plain APK, no Google account
    - pkgs.tailscale.com/stable/#android
    - Mirrored on Tailscale Android GitHub releases
    - F-Droid builds from same source
- Memory: roughly 10–15 MB RAM idle on mobile
    - iOS profiling: capped at 15 MB, sat around 13
    - Android a bit higher, well under 20 MB
    - Rabbit R1 has around 4 GB RAM

### Authentication without Google account

- "Sign in with other" supports any SSO provider
- Auth key from admin console for headless or kiosk setup
- QR code flow approved from another machine

### File transfer layer

- Tailscale only provides the network tunnel
- Still need receiving endpoint
    - SFTP server, SMB share or simple upload endpoint

### Install and authenticate over USB-C

- PC server app already connected to Rabbit by USB-C
- Install and authenticate both over that cable
    - Enable developer options, USB debugging
    - Confirm with adb devices, fetch official APK
- Speaker already knows these steps

### Tailscale confirmed as better fit

- More secure than Cloudflare tunnel
- Cloudflare: tunnel daemon, Cloudflare as middleman
- Tailscale: WireGuard mesh, end-to-end, nothing exposed
- Simpler, lighter, more secure for one recorder to one machine

### Multiple instances on one computer

- Possible; each needs own state directory and socket path
- Linux: separate processes with state and socket flags
    - Each registers as own node with own IP
    - Tag differently or use separate tailnets
- Windows: no native multi-instance support
    - Use separate containers or WSL instances

### Tailscale Serve

- Built-in feature to serve a web page from desktop
    - Run tiny web server, e.g. Python HTTP server or Nginx
    - Command like "tailscale serve 8080"
- Reachable by Tailscale IP or MagicDNS, tailnet devices only
- Funnel for public access; Serve right for this use

## Whiteboard animation

### Name of the style

- Hand drawing on whiteboard with background narrator
- Called whiteboard animation, sketch animation or doodle video
- Names: VideoScribe, Doodly, Common Craft
    - Common Craft pioneered the explainer style
    - RSA Animate: more artistic, fast-paced sketches
- Speaker: RSA Animate does a really good job

### RSA Animate style with AI

- Want drawing on whiteboard; hand and pencil not needed
- Existing tools, nothing custom needed
    - Ancon AI: writes narration, draws stroke by stroke, syncs voice
        - No-hand mode
    - animation.ai: animates existing images, outline-first, no-hand option
- Mostly subscription-based
    - Ancon AI, Colpo, Animaker on monthly plans

### Building a custom tool

- Wants to build own tool, or assemble open source parts
- Access to Claude and Codex to drive frontier AIs
- Not commercial; for own videos on speaker's YouTube account

### Open source pipeline

- Whiteboard-Animator on GitHub as drawing engine
    - MIT licensed; CPU only; no GPU or API keys for core
    - Traces strokes, writes text word by word, brush fills
    - Paces to narration audio
    - Does not generate images
- Proposed pipeline
    - Claude or Codex writes scene-by-scene script
        - One idea per scene, about 12–15 words, visual description
    - Image model draws black-ink-on-white illustration
        - Flux or GPT Image
    - Whiteboard-Animator animates each image with narration clip
    - FFmpeg stitches scenes
- Narration: ElevenLabs or free Edge TTS
- HandDraw-Skill: near end-to-end alternative
    - SVG, Remotion, FFmpeg, Edge TTS
    - Built to be driven by Claude Code or Codex
    - Could fork and restyle toward RSA Animate

### How it chooses what to draw first

- Question: how it knows to draw the apple when narration mentions it
- No semantic matching with the script
- Spatial heuristics
    - Containers before contents, shapes before labels
    - Text in reading order; dots attach to glyphs
    - Likely left to right, top to bottom
- Override with region plan JSON listing reveal order
- Optional Gemini step proposes regions and order from image plus narration
- Smart part is a one-time LLM annotation pass

### Raster mask or vector paths

- Raster, not vector; no SVG or path tracing
- Finds connected blobs darker than about 240 gray
- Small text detector marks words, written left to right
- Skeleton (centerline) per component; reveal from real endpoint
    - Closed shapes: single front around boundary
    - Fills: outline pass, then angled sweep or bristle strokes
    - Junctions (X, grid) split into sequential pen paths
- Each pixel gets reveal timestamp; only active pixels fade-blend
- Streams out through FFmpeg
- Works only on clean ink-on-white; photos, gradients, textures fail

### AI video approach

- Brainstorm: prompt a video model with drawing order
    - Apple first, shade red; pear, golden yellow; zoom to orange last
- VideoSketcher research project
    - LLM splits subject into parts and orders them
        - Example: cake base, frosting, candles
    - Video diffusion model trained on artist SVGs
    - Research prototype; semantic, not pixel-precise control
- Creators' practice: start and end frame trick
    - Keyframes: blank board, apple outline, shaded apple, pear added
    - Chained through Seedance or Veo to interpolate motion
    - Kling motion brush animates only painted region
- Nobody has nailed script-driven stroke-level control with video models

### Practical takeaway and hybrid

- Precise order: deterministic renderer plus LLM region planning
- Organic feel: video model, chained keyframes, babysitting order
- Common hybrid
    - Image model for illustration
    - Stroke renderer for drawing motion
    - Video model only for camera moves or shading transitions
    - Roughly 80% of the RSA Animate look

### Giant canvas plus camera moves

- Full RSA-style board: maybe 30–40 drawings, zoomed out at end
- Camera starts top left, pans, scans, zooms in and out
- Speaker's idea: render super high resolution first, add camera from narration second
- Grok: how real RSA Animate was built
    - Cognitive worked on canvas several times frame size
    - Camera a separate moving layer
- Render full board at 4K or 8K, drawings in final positions
- Camera path keyframes with timestamps, driven by narration
    - Example: 3 s zoom top left at 200%; 7 s pan to apple; 12 s zoom out
- Remotion handles this natively: animate transform on a big image
- Speaker already knows Remotion

### Generating the large canvas

- Grid of tiles at normal resolution, e.g. 1024×1024
- Prompts describe each region and what bleeds into neighbors
- Composite with small overlap and soft edge blend
- Result: one canvas, maybe 8K or 16K
- Feed each tile its neighbors' context so drawings aren't cut

### Finished render first

- Speaker asks: finished render created first?
- Yes: whole board, all 30–40 drawings, inked and shaded, as one static image
- Remotion animates virtual camera that pans, zooms and reveals, synced to narration
- Motion comes from camera revealing existing content, not live strokes
