Sep 24, 17:03
Session plan
- Discussion with Grok on two subjects
- Building a server inside R1CORD
- Research on chalkboard-style rapid viz: animated drawing, hand, pencils
R1CORD server connectivity
Current setup
- Desktop server app talks to custom-programmed Rabbit
- Rabbit sends files to desktop over the internet
- Routed through a Cloudflare tunnel
- Tunnel avoids opening ports on home network
- Rabbit needs only outbound access to tunnel endpoint
Goal: drop Cloudflare
- Server secure enough to not need Cloudflare
- Asking users to configure a Cloudflare tunnel is not user-friendly
- Run server as a service, locked down for the Rabbit’s user
- Typically a one-to-one setup
Mesh VPN suggestion first rejected
- Grok suggests Tailscale or ZeroTier for one-to-one setups
- Speaker rejects: wants only an installable PC executable plus APK
- No full configuration mode if avoidable
Opening a port without the router
- Not possible from the desktop alone
- Router translates private IP to public; inbound must be forwarded
- UPnP closest option: app asks router to open port
- Often disabled; some ISPs block it; not guaranteed
How Cloudflare Tunnel works
- Does not bypass the router
- Desktop makes outbound connection to Cloudflare edge servers
- Routers almost always allow outbound traffic
- Incoming requests to Cloudflare hostname return down the same pipe
- Long-lived outbound connection, multiplexed return traffic
How Dropbox works
- Desktop client connects outbound and waits
- Files live on Dropbox servers, not the desktop
- Dropbox is the server; desktop is just a client
- Cloudflare: desktop is server, Cloudflare the middleman
- Own-hardware option needs a relay both sides reach outbound
Google Drive or Dropbox as relay
- Shared folder as meeting point
- Rabbit uploads to designated folder
- Desktop watches via API or local sync
- No server, tunnel or router config
- Trade-offs: latency, API limits, no real-time push
- Speaker: real-time push not needed
Privacy requirement
- Main worry is privacy
- R1CORD: converted Rabbit as real-time audio recorder
- Conversations should not sit on someone else’s cloud server
- Drive/Dropbox question only to understand possible architectures
Own PHP server
- Own PHP server on cPanel would work
- Most people don’t have one
Tailscale
How it works
- Mesh VPN built on WireGuard
- Tiny client on Rabbit and desktop
- Coordination servers only exchange keys and find peers
- Audio flows directly, end-to-end encrypted
- Falls back to encrypted relays if no direct path
- Desktop server listens on Tailscale IP; Rabbit connects there
- No router config, port forwarding, Cloudflare or third-party storage
- Free tier: 100 devices, three users
- Cost: install and log in once on both ends
Tailscale vs Cloudflare
- Cloudflare in the middle of every packet
- Terminates connection at its edge
- Acts as gatekeeper for the service
- Service reachable from whole internet via hostname
- Tailscale peer-to-peer by default
- Out of the data path after handshake
- Never holds the keys
- Private network of only the two devices; no URL
Security vs ease of setup
- Security: Tailscale wins clearly
- Matters for a product sold on private conversations
- Setup: Cloudflare simpler on paper
- One desktop install, tunnel token, Rabbit hits a URL
- Requires managing tunnel credentials and hostname
- Tailscale close: one-click desktop, app plus login on Rabbit
- Maybe two extra taps; one-time
- Grok’s pick for R1CORD: Tailscale
Rabbit constraints
- Concern about memory footprint and CPU on underpowered Rabbit
- Hacked Rabbit runs minimal Android, probably 13
- No Play Store access
- Questions: sideload without Play Store? Open source?
Interruption and restated question
- Repeated “Hey, Grok” attempts; connection interrupted
- Rabbit as dedicated offline recorder, uploads when online
- No Google account on device
- Can Tailscale go inside own APK or install as its own APK?
Open source, sideloading, memory
- Fully open source; installs as plain APK, no Google account
- Memory: roughly 10–15 MB RAM idle on mobile
- iOS profiling: capped at 15 MB, sat around 13
- Android a bit higher, well under 20 MB
- Rabbit R1 has around 4 GB RAM
Authentication without Google account
- “Sign in with other” supports any SSO provider
- Auth key from admin console for headless or kiosk setup
- QR code flow approved from another machine
File transfer layer
- Tailscale only provides the network tunnel
- Still need receiving endpoint
- SFTP server, SMB share or simple upload endpoint
Install and authenticate over USB-C
- PC server app already connected to Rabbit by USB-C
- Install and authenticate both over that cable
- Enable developer options, USB debugging
- Confirm with adb devices, fetch official APK
- Speaker already knows these steps
Tailscale confirmed as better fit
- More secure than Cloudflare tunnel
- Cloudflare: tunnel daemon, Cloudflare as middleman
- Tailscale: WireGuard mesh, end-to-end, nothing exposed
- Simpler, lighter, more secure for one recorder to one machine
Multiple instances on one computer
- Possible; each needs own state directory and socket path
- Linux: separate processes with state and socket flags
- Each registers as own node with own IP
- Tag differently or use separate tailnets
- Windows: no native multi-instance support
- Use separate containers or WSL instances
Tailscale Serve
- Built-in feature to serve a web page from desktop
- Run tiny web server, e.g. Python HTTP server or Nginx
- Command like “tailscale serve 8080”
- Reachable by Tailscale IP or MagicDNS, tailnet devices only
- Funnel for public access; Serve right for this use
Whiteboard animation
Name of the style
- Hand drawing on whiteboard with background narrator
- Called whiteboard animation, sketch animation or doodle video
- Names: VideoScribe, Doodly, Common Craft
- Common Craft pioneered the explainer style
- RSA Animate: more artistic, fast-paced sketches
- Speaker: RSA Animate does a really good job
RSA Animate style with AI
- Want drawing on whiteboard; hand and pencil not needed
- Existing tools, nothing custom needed
- Ancon AI: writes narration, draws stroke by stroke, syncs voice
- animation.ai: animates existing images, outline-first, no-hand option
- Mostly subscription-based
- Ancon AI, Colpo, Animaker on monthly plans
- Wants to build own tool, or assemble open source parts
- Access to Claude and Codex to drive frontier AIs
- Not commercial; for own videos on speaker’s YouTube account
Open source pipeline
- Whiteboard-Animator on GitHub as drawing engine
- MIT licensed; CPU only; no GPU or API keys for core
- Traces strokes, writes text word by word, brush fills
- Paces to narration audio
- Does not generate images
- Proposed pipeline
- Claude or Codex writes scene-by-scene script
- One idea per scene, about 12–15 words, visual description
- Image model draws black-ink-on-white illustration
- Whiteboard-Animator animates each image with narration clip
- FFmpeg stitches scenes
- Narration: ElevenLabs or free Edge TTS
- HandDraw-Skill: near end-to-end alternative
- SVG, Remotion, FFmpeg, Edge TTS
- Built to be driven by Claude Code or Codex
- Could fork and restyle toward RSA Animate
How it chooses what to draw first
- Question: how it knows to draw the apple when narration mentions it
- No semantic matching with the script
- Spatial heuristics
- Containers before contents, shapes before labels
- Text in reading order; dots attach to glyphs
- Likely left to right, top to bottom
- Override with region plan JSON listing reveal order
- Optional Gemini step proposes regions and order from image plus narration
- Smart part is a one-time LLM annotation pass
Raster mask or vector paths
- Raster, not vector; no SVG or path tracing
- Finds connected blobs darker than about 240 gray
- Small text detector marks words, written left to right
- Skeleton (centerline) per component; reveal from real endpoint
- Closed shapes: single front around boundary
- Fills: outline pass, then angled sweep or bristle strokes
- Junctions (X, grid) split into sequential pen paths
- Each pixel gets reveal timestamp; only active pixels fade-blend
- Streams out through FFmpeg
- Works only on clean ink-on-white; photos, gradients, textures fail
AI video approach
- Brainstorm: prompt a video model with drawing order
- Apple first, shade red; pear, golden yellow; zoom to orange last
- VideoSketcher research project
- LLM splits subject into parts and orders them
- Example: cake base, frosting, candles
- Video diffusion model trained on artist SVGs
- Research prototype; semantic, not pixel-precise control
- Creators’ practice: start and end frame trick
- Keyframes: blank board, apple outline, shaded apple, pear added
- Chained through Seedance or Veo to interpolate motion
- Kling motion brush animates only painted region
- Nobody has nailed script-driven stroke-level control with video models
Practical takeaway and hybrid
- Precise order: deterministic renderer plus LLM region planning
- Organic feel: video model, chained keyframes, babysitting order
- Common hybrid
- Image model for illustration
- Stroke renderer for drawing motion
- Video model only for camera moves or shading transitions
- Roughly 80% of the RSA Animate look
Giant canvas plus camera moves
- Full RSA-style board: maybe 30–40 drawings, zoomed out at end
- Camera starts top left, pans, scans, zooms in and out
- Speaker’s idea: render super high resolution first, add camera from narration second
- Grok: how real RSA Animate was built
- Cognitive worked on canvas several times frame size
- Camera a separate moving layer
- Render full board at 4K or 8K, drawings in final positions
- Camera path keyframes with timestamps, driven by narration
- Example: 3 s zoom top left at 200%; 7 s pan to apple; 12 s zoom out
- Remotion handles this natively: animate transform on a big image
- Speaker already knows Remotion
Generating the large canvas
- Grid of tiles at normal resolution, e.g. 1024×1024
- Prompts describe each region and what bleeds into neighbors
- Composite with small overlap and soft edge blend
- Result: one canvas, maybe 8K or 16K
- Feed each tile its neighbors’ context so drawings aren’t cut
Finished render first
- Speaker asks: finished render created first?
- Yes: whole board, all 30–40 drawings, inked and shaded, as one static image
- Remotion animates virtual camera that pans, zooms and reveals, synced to narration
- Motion comes from camera revealing existing content, not live strokes