Sep 24, 17:03

Session plan

  • Discussion with Grok on two subjects
    • Building a server inside R1CORD
    • Research on chalkboard-style rapid viz: animated drawing, hand, pencils

R1CORD server connectivity

Current setup

  • Desktop server app talks to custom-programmed Rabbit
  • Rabbit sends files to desktop over the internet
  • Routed through a Cloudflare tunnel
  • Tunnel avoids opening ports on home network
  • Rabbit needs only outbound access to tunnel endpoint

Goal: drop Cloudflare

  • Server secure enough to not need Cloudflare
  • Asking users to configure a Cloudflare tunnel is not user-friendly
  • Run server as a service, locked down for the Rabbit’s user
  • Typically a one-to-one setup

Mesh VPN suggestion first rejected

  • Grok suggests Tailscale or ZeroTier for one-to-one setups
  • Speaker rejects: wants only an installable PC executable plus APK
  • No full configuration mode if avoidable

Opening a port without the router

  • Not possible from the desktop alone
  • Router translates private IP to public; inbound must be forwarded
  • UPnP closest option: app asks router to open port
    • Often disabled; some ISPs block it; not guaranteed

How Cloudflare Tunnel works

  • Does not bypass the router
  • Desktop makes outbound connection to Cloudflare edge servers
  • Routers almost always allow outbound traffic
  • Incoming requests to Cloudflare hostname return down the same pipe
  • Long-lived outbound connection, multiplexed return traffic

How Dropbox works

  • Desktop client connects outbound and waits
  • Files live on Dropbox servers, not the desktop
  • Dropbox is the server; desktop is just a client
  • Cloudflare: desktop is server, Cloudflare the middleman
  • Own-hardware option needs a relay both sides reach outbound

Google Drive or Dropbox as relay

  • Shared folder as meeting point
    • Rabbit uploads to designated folder
    • Desktop watches via API or local sync
  • No server, tunnel or router config
  • Trade-offs: latency, API limits, no real-time push
  • Speaker: real-time push not needed

Privacy requirement

  • Main worry is privacy
  • R1CORD: converted Rabbit as real-time audio recorder
  • Conversations should not sit on someone else’s cloud server
  • Drive/Dropbox question only to understand possible architectures

Own PHP server

  • Own PHP server on cPanel would work
  • Most people don’t have one

Tailscale

How it works

  • Mesh VPN built on WireGuard
  • Tiny client on Rabbit and desktop
  • Coordination servers only exchange keys and find peers
  • Audio flows directly, end-to-end encrypted
  • Falls back to encrypted relays if no direct path
  • Desktop server listens on Tailscale IP; Rabbit connects there
  • No router config, port forwarding, Cloudflare or third-party storage
  • Free tier: 100 devices, three users
  • Cost: install and log in once on both ends

Tailscale vs Cloudflare

  • Cloudflare in the middle of every packet
    • Terminates connection at its edge
    • Acts as gatekeeper for the service
    • Service reachable from whole internet via hostname
  • Tailscale peer-to-peer by default
    • Out of the data path after handshake
    • Never holds the keys
    • Private network of only the two devices; no URL

Security vs ease of setup

  • Security: Tailscale wins clearly
    • Matters for a product sold on private conversations
  • Setup: Cloudflare simpler on paper
    • One desktop install, tunnel token, Rabbit hits a URL
    • Requires managing tunnel credentials and hostname
  • Tailscale close: one-click desktop, app plus login on Rabbit
    • Maybe two extra taps; one-time
  • Grok’s pick for R1CORD: Tailscale

Rabbit constraints

  • Concern about memory footprint and CPU on underpowered Rabbit
  • Hacked Rabbit runs minimal Android, probably 13
  • No Play Store access
  • Questions: sideload without Play Store? Open source?

Interruption and restated question

  • Repeated “Hey, Grok” attempts; connection interrupted
  • Rabbit as dedicated offline recorder, uploads when online
  • No Google account on device
  • Can Tailscale go inside own APK or install as its own APK?

Open source, sideloading, memory

  • Fully open source; installs as plain APK, no Google account
  • Memory: roughly 10–15 MB RAM idle on mobile
    • iOS profiling: capped at 15 MB, sat around 13
    • Android a bit higher, well under 20 MB
    • Rabbit R1 has around 4 GB RAM

Authentication without Google account

  • “Sign in with other” supports any SSO provider
  • Auth key from admin console for headless or kiosk setup
  • QR code flow approved from another machine

File transfer layer

  • Tailscale only provides the network tunnel
  • Still need receiving endpoint
    • SFTP server, SMB share or simple upload endpoint

Install and authenticate over USB-C

  • PC server app already connected to Rabbit by USB-C
  • Install and authenticate both over that cable
    • Enable developer options, USB debugging
    • Confirm with adb devices, fetch official APK
  • Speaker already knows these steps

Tailscale confirmed as better fit

  • More secure than Cloudflare tunnel
  • Cloudflare: tunnel daemon, Cloudflare as middleman
  • Tailscale: WireGuard mesh, end-to-end, nothing exposed
  • Simpler, lighter, more secure for one recorder to one machine

Multiple instances on one computer

  • Possible; each needs own state directory and socket path
  • Linux: separate processes with state and socket flags
    • Each registers as own node with own IP
    • Tag differently or use separate tailnets
  • Windows: no native multi-instance support
    • Use separate containers or WSL instances

Tailscale Serve

  • Built-in feature to serve a web page from desktop
    • Run tiny web server, e.g. Python HTTP server or Nginx
    • Command like “tailscale serve 8080”
  • Reachable by Tailscale IP or MagicDNS, tailnet devices only
  • Funnel for public access; Serve right for this use

Whiteboard animation

Name of the style

  • Hand drawing on whiteboard with background narrator
  • Called whiteboard animation, sketch animation or doodle video
  • Names: VideoScribe, Doodly, Common Craft
    • Common Craft pioneered the explainer style
    • RSA Animate: more artistic, fast-paced sketches
  • Speaker: RSA Animate does a really good job

RSA Animate style with AI

  • Want drawing on whiteboard; hand and pencil not needed
  • Existing tools, nothing custom needed
    • Ancon AI: writes narration, draws stroke by stroke, syncs voice
      • No-hand mode
    • animation.ai: animates existing images, outline-first, no-hand option
  • Mostly subscription-based
    • Ancon AI, Colpo, Animaker on monthly plans

Building a custom tool

  • Wants to build own tool, or assemble open source parts
  • Access to Claude and Codex to drive frontier AIs
  • Not commercial; for own videos on speaker’s YouTube account

Open source pipeline

  • Whiteboard-Animator on GitHub as drawing engine
    • MIT licensed; CPU only; no GPU or API keys for core
    • Traces strokes, writes text word by word, brush fills
    • Paces to narration audio
    • Does not generate images
  • Proposed pipeline
    • Claude or Codex writes scene-by-scene script
      • One idea per scene, about 12–15 words, visual description
    • Image model draws black-ink-on-white illustration
      • Flux or GPT Image
    • Whiteboard-Animator animates each image with narration clip
    • FFmpeg stitches scenes
  • Narration: ElevenLabs or free Edge TTS
  • HandDraw-Skill: near end-to-end alternative
    • SVG, Remotion, FFmpeg, Edge TTS
    • Built to be driven by Claude Code or Codex
    • Could fork and restyle toward RSA Animate

How it chooses what to draw first

  • Question: how it knows to draw the apple when narration mentions it
  • No semantic matching with the script
  • Spatial heuristics
    • Containers before contents, shapes before labels
    • Text in reading order; dots attach to glyphs
    • Likely left to right, top to bottom
  • Override with region plan JSON listing reveal order
  • Optional Gemini step proposes regions and order from image plus narration
  • Smart part is a one-time LLM annotation pass

Raster mask or vector paths

  • Raster, not vector; no SVG or path tracing
  • Finds connected blobs darker than about 240 gray
  • Small text detector marks words, written left to right
  • Skeleton (centerline) per component; reveal from real endpoint
    • Closed shapes: single front around boundary
    • Fills: outline pass, then angled sweep or bristle strokes
    • Junctions (X, grid) split into sequential pen paths
  • Each pixel gets reveal timestamp; only active pixels fade-blend
  • Streams out through FFmpeg
  • Works only on clean ink-on-white; photos, gradients, textures fail

AI video approach

  • Brainstorm: prompt a video model with drawing order
    • Apple first, shade red; pear, golden yellow; zoom to orange last
  • VideoSketcher research project
    • LLM splits subject into parts and orders them
      • Example: cake base, frosting, candles
    • Video diffusion model trained on artist SVGs
    • Research prototype; semantic, not pixel-precise control
  • Creators’ practice: start and end frame trick
    • Keyframes: blank board, apple outline, shaded apple, pear added
    • Chained through Seedance or Veo to interpolate motion
    • Kling motion brush animates only painted region
  • Nobody has nailed script-driven stroke-level control with video models

Practical takeaway and hybrid

  • Precise order: deterministic renderer plus LLM region planning
  • Organic feel: video model, chained keyframes, babysitting order
  • Common hybrid
    • Image model for illustration
    • Stroke renderer for drawing motion
    • Video model only for camera moves or shading transitions
    • Roughly 80% of the RSA Animate look

Giant canvas plus camera moves

  • Full RSA-style board: maybe 30–40 drawings, zoomed out at end
  • Camera starts top left, pans, scans, zooms in and out
  • Speaker’s idea: render super high resolution first, add camera from narration second
  • Grok: how real RSA Animate was built
    • Cognitive worked on canvas several times frame size
    • Camera a separate moving layer
  • Render full board at 4K or 8K, drawings in final positions
  • Camera path keyframes with timestamps, driven by narration
    • Example: 3 s zoom top left at 200%; 7 s pan to apple; 12 s zoom out
  • Remotion handles this natively: animate transform on a big image
  • Speaker already knows Remotion

Generating the large canvas

  • Grid of tiles at normal resolution, e.g. 1024×1024
  • Prompts describe each region and what bleeds into neighbors
  • Composite with small overlap and soft edge blend
  • Result: one canvas, maybe 8K or 16K
  • Feed each tile its neighbors’ context so drawings aren’t cut

Finished render first

  • Speaker asks: finished render created first?
  • Yes: whole board, all 30–40 drawings, inked and shaded, as one static image
  • Remotion animates virtual camera that pans, zooms and reveals, synced to narration
  • Motion comes from camera revealing existing content, not live strokes