Sep 24, 17:03
2026-09-24 16:03 · 38:41
Hey!
Okay, so we’re going to have a discussion with Grok, and we’re going to talk about
a couple different things.
Specifically, one is going to be Grok’s
or the ability for us to build a server inside of Record.
And the second is going to be
the
some research on
how we can do
kind of a chalkboard style
rapid viz, you know, kind of an animated drawing with a hand and the
pencils and stuff like that.
So,
so let’s start off with the first one.
Okay, Grok.
I want to talk about
my Record server.
I have a server app that fits on a desktop
that communicates with my rabbit that I custom programmed
to be able to send files to my desktop.
and I do it through CloudFlare and a tunnel approach and it’s done from the internet.
So, tell me you understand what I’m talking about.
Yeah, I follow you.
You’ve got a rabbit R1 you’ve custom programmed to push files out over the internet.
And instead of exposing your desktop directly, you’re routing it through a CloudFlare tunnel.
So the connection comes in securely without opening ports on your home network.
The desktop runs a server app that receives those files on the other end.
That’s a solid setup.
CloudFlare tunnel handles the ingress, keeps your machine off the public internet,
and the rabbit just needs outbound access to reach the tunnel endpoint.
What do you want to dig into?
The rabbit side?
The tunnel config?
The server app?
Or something else?
So I want to talk about the server app.
And I want to think, I’d like to make this so that I could,
so the server would be secure enough to not require CloudFlare.
Because asking a user to configure CloudFlare to create a tunnel for them
is kind of ridiculous in terms of user-friendliness.
And so I’d like, ideally, to be able to just run the server as a service as it is now
and somehow lock it down enough that people can, that a user,
that the user of the rabbit can get to it.
Typically, this is a one-to-one deal.
Tunnels are powerful, but they’re a pain to set up for a regular person.
For a one-to-one setup like yours,
something like tail scale or zero tier is probably the sweet spot.
You install a tiny client on the rabbit, run a node on the desktop,
and you just see…
No, no, I don’t want to do that either.
So it’s just, it’s got to be,
you got to basically have the executable on the PC
that you can install and run.
And the rabbit is just an APK, right?
It’s just an APK.
So it’s not, I don’t want to go through a complete configuration mode if I don’t have to.
Now, let’s back up the situation.
Is it possible to open a port on my desktop without having to go through my router,
my Wi-Fi router, or whatever?
Is it possible to open up a port on my desktop?
Is there a program like, I don’t know?
Short answer, no.
Not from inside the desktop alone.
Your desktop sits behind the router,
and the router is doing that.
It translates your private IP to the public one the internet sees.
So any incoming connection has to hit the router first,
and the router has to know to forward it to your desktop.
There’s no way for the desktop to punch through that on its own.
The closest thing to just run it without manual router config is UPnP.
The app asks the router to open the port automatically.
Most home routers support it,
but a lot of people have it disabled or don’t know it’s on.
It’s not guaranteed, and some ISPs block it entirely.
What about, what if you truly…
So tell me what Cloudflare does then,
because I don’t mess with the router that I’m aware of.
Cloudflare just bypasses the router somehow.
How does that work?
Cloudflare doesn’t bypass your router at all.
Your router still has to allow the outbound connection.
What Cloudflare does is flip the direction.
Instead of the internet reaching into your network,
your desktop reaches out to Cloudflare’s edge servers first.
An outbound connection.
Routers almost always allow outbound traffic,
so nothing on the router needs to change.
Once that connection is up,
Cloudflare’s edge holds it open,
and any incoming request that hits your Cloudflare hostname
gets routed back down
through that same outbound pipe.
So the tunnel is really just a long-lived outbound connection
that Cloudflare multiplexes return traffic over.
That’s why you never touch the router.
You’re not opening anything inbound.
You’re just making a normal outbound call,
and Cloudflare does the rest on their side.
So how does a program like Dropbox,
or, you know, how does it handle it?
Because, you know, I’m basically accessing files on the Dropbox server
without having to set up a Cloudflare or any other tunneling software.
Dropbox flips the model completely.
Your desktop never receives an inbound connection.
Instead, the Dropbox client on your PC makes an outbound connection to Dropbox’s servers
and just sits there waiting.
When you want a file,
your phone or another device talks to Dropbox’s cloud,
and Dropbox pushes the file down to your desktop
through that already open outbound pipe.
The key difference is where the files live.
They’re on Dropbox’s servers, not on your machine.
So there’s no server running on your desktop
that the outside world needs to reach.
Your desktop is just a client pulling and pushing data
to a central server that everyone else also talks to.
That’s the fundamental trade-off.
With Cloudflare Tunnel,
your desktop is the server,
and Cloudflare is the middleman.
With Dropbox,
Dropbox is the server,
and your desktop is just another client.
If you want the files to stay on your own hardware,
but still be reachable without touching the router,
you’d need something in between,
a relay that both sides connect outbound to,
kind of like a mini Dropbox,
but pointed at your machine instead of a cloud store.
So, what if I were to configure…
Go ahead.
Can I configure any…
Just for the sake of discussion,
is there a way to configure Google Drive
to handle any of this?
Like, can I use, like,
ACK files on Google Drive,
or…
or…
even ACK files on Dropbox
to do
any of this?
Yeah, absolutely.
And it’s actually a really clean fit for what you’re doing.
You could treat a shared Drive or Dropbox folder as your relay.
The Rabbit uploads files to a designated folder.
Your desktop app watches that folder,
either through the Drive or Dropbox API,
or even just by syncing the folder locally
and picking up new files as they land.
No server,
no tunnel,
no router config.
Both sides just talk outbound to Google or Dropbox,
and the folder is the meeting point.
The trade-offs are latency and API limits.
You’re not getting a real-time push.
You’re pulling or running on syncing.
I don’t need the real-time push.
I don’t need that.
That’s not something that I’m worried about.
But,
here’s what I am worried about.
It’s privacy, right?
So, the whole point of this product
is that you can have discussions
with
this converted rabbit.
It’s an audio…
It’s a real-time audio recorder, right?
So, that’s what it does.
It’s R1 Cord,
is what the name of the product is.
You can look it up.
But,
the point of it is that
you just don’t want
your conversations
in the cloud
on somebody else’s server.
So,
I understand.
I asked that question
about
Dropbox
and Google Drive
because I just wanted to understand
a little bit about how,
you know,
what kind of architectures
may work
or may not work.
So,
what about
if you had your own,
let’s see,
what if you had your,
your own PHP server
like on a cPanel
or something like that,
you could obviously do it there,
but most people don’t have that.
I’m trying to think of
what most people have
that they can,
that they can use.
I don’t really have a…
Let’s back up.
Let’s go and talk about
Tailscale now.
Tailscale is worth a serious look
for exactly your privacy concern.
It’s a mesh VPN built on WireGuard,
and the privacy story
is actually pretty good
for your use case.
Both the Rabbit
and your desktop
install a tiny Tailscale client.
On Android,
that’s straightforward,
and on the desktop,
it’s one installer.
They both authenticate
to Tailscale’s coordination servers
just long enough
to exchange keys
and find each other.
After that,
the actual audio data
flows directly
between the two devices,
encrypted end-to-end.
Tailscale servers
never see your conversations.
They only know
that two nodes are connected.
If a direct peer-to-peer path
isn’t possible,
it falls back
to encrypted relay servers.
But even then,
the data is encrypted
and Tailscale can’t read it.
The big win for you
is that your desktop server app
just listens
on the Tailscale IP address,
and the Rabbit
connects to that same address.
No router config,
no port forwarding,
no Cloudflare,
no third-party file storage
holding your audio.
The desktop is the server,
the Rabbit is the client,
and Tailscale is just
the private network
pipe between them.
And the free tier
covers up to 100 devices
and three users,
which is more than enough
for a one-to-one product
like R1 Cord.
The one thing to think through
is the initial setup experience.
The user has to install Tailscale
on both ends
and log in once.
That’s a couple of extra steps
compared to just installing an APK.
But it’s a one-time thing,
and it’s about as painless
as mesh networking gets.
Want to dig into
how the Rabbit side
would actually connect,
or the desktop server config?
No, I want to talk
a little bit more
about what the difference
between Tailscale
and Cloudflare is.
The core difference
is architecture
and who you’re trusting.
With Cloudflare Tunnel,
your desktop connects
outbound to Cloudflare’s edge,
and every request
from the Rabbit
has to pass through
Cloudflare servers
on the way in
and back out.
Cloudflare is sitting
in the middle
of every single packet.
Even though the connection
to Cloudflare is encrypted,
Cloudflare is the one
terminating that connection
at their edge.
They’re the one deciding
what gets forwarded
to your machine.
So you’re trusting Cloudflare
not just with routing,
but with being the gatekeeper
for your service.
Tailscale works differently.
It’s peer-to-peer
by default.
The Rabbit
and your desktop
find each other,
usually directly
over the internet,
sometimes through a relay
if a direct path
is impossible.
But the actual data
is encrypted end-to-end
between the two devices.
Tailscale’s coordination
servers only handle
the initial handshake,
exchanging keys,
figuring out
how the two nodes
can reach each other.
After that,
Tailscale is out
of the data path entirely.
They can’t read your audio
even if they wanted to
because they never
have the keys.
So the practical difference
for your product,
with Cloudflare,
you’re building on top
of someone else’s
public-facing infrastructure,
and your service
is reachable
from the entire internet
through that
Cloudflare host name.
With Tailscale,
you’re building
a private network
that only your two devices
are part of.
Nothing is exposed
to the public internet
at all.
The Rabbit
connects to a Tailscale IP,
not a public domain.
There’s no URL.
So tell me,
one,
which is more secure,
and number two,
which is easier
to set up
for the user?
For security,
Tailscale wins,
and it’s not close
for your use case.
With Tailscale,
the data path
is end-to-end encrypted
between the Rabbit
and your desktop.
Tailscale servers
never touch your audio.
With Cloudflare,
Cloudflare’s edge
is actively proxying
every request.
So even though
the tunnel itself
is encrypted,
Cloudflare is a trusted
intermediary in the middle.
For a product
whose whole selling point
is that your conversations
stay private,
that distinction
matters a lot.
For ease of setup,
Cloudflare is simpler
on paper.
You install one thing
on the desktop,
get a tunnel token,
and you’re done.
The Rabbit just hits
a URL.
But Tailscale
has gotten really close.
The desktop install
is one click,
the Rabbit just needs
the Tailscale app
from the Play Store
and a login,
and after that,
it’s automatic.
The difference is maybe
two extra taps
on the Rabbit side.
Honestly,
for a non-technical user,
both are doable,
but Tailscale’s setup
is more of a login
once and forget it experience,
whereas Cloudflare requires
you to manage tunnel credentials
and a hostname.
So if I had to pick
for R1 Cord,
Tailscale gives you
better security
with nearly the same
setup effort.
The only real cost
is that extra login
step on the Rabbit,
and that’s a one-time thing.
Yeah,
so let’s talk about that
for a couple reasons.
One is I want to understand
the memory footprint
of it,
and if it’s going to
chew up any cycles,
because that’s an underpowered
device,
the Rabbit is.
And number two is
this is a very minimum,
I think it’s an Android 12,
maybe,
or 13 OS,
that goes on the Rabbit
that you put on the Rabbit.
That’s for hacking the Rabbit.
I think it’s maybe Android 13
that they give you
to hack it with.
And it’s got to work with that.
And it’s not,
you can’t go to a Play Store
to download it.
So can you download Tailscale
and install it
without going to the Play Store?
is it open source?
Hey, Grok.
Hey, Grok.
Hey, Grok.
So we were talking about
Tailscale versus Cloudflare,
and I’ve got a Rabbit
that we’ve hacked
that has a recording app,
and I’m trying to send data
from the Rabbit
using Android 13.
And my own APK,
it’s the only device,
and I want to understand,
one,
the memory footprint
of Tailscale on the Rabbit,
which is what you recommended
we use Tailscale
instead of Cloudflare,
what the memory footprint is,
and how we install it
on the Rabbit
if we don’t have the Play Store
and we don’t have
a user ID
on the Rabbit.
The Rabbit is just going to be
this little utility device
that’s just a recorder,
just an offline recorder,
and when it goes online,
it’ll upload its files.
So I’m trying to understand
is Tailscale
is that open source?
If so,
can I put it
inside my APK,
or do I,
does it have its own APK
that I can install
without having to go through
the Play Store?
Hey, Grok.
Hey, Grok.
Okay, Grok.
Can you hear me?
Can you hear me?
Hello?
Can you hear me?
Hello?
Hey, I’m here.
What can I help you with?
Okay, we were just talking
and got interrupted
to get some kind of issue,
but we were talking about
Tailscale on the Rabbit device
when you convert a Rabbit
to an Android 13 machine.
what is the memory footprint
of Tailscale?
And the machine I’m putting on
is a dedicated voice recorder,
offline, online voice recorder,
and I want it
to be able to upload files
to my computer at home.
And I want to understand
how I install,
if I don’t have a Google account
to install from
on the device,
or can I just,
is Tailscale open source?
And if so,
can I just install it
as an APK directly?
Let me check the latest
on Tailscale’s footprint
and sideloading.
Yes,
Tailscale is fully open source,
and you can absolutely install it
as a plain APK
with no Google account involved.
Official stable APKs
are at
slash stable
slash hash android.
And the same builds
are mirrored
on the Tailscale Android
GitHub releases page,
so you can grab them
with Uptanium
or just sideload directly.
FDroid also builds
from the same source
if you prefer that route.
On memory,
Tailscale’s baseline
on mobile
is roughly 10 to 15 megabytes
of RAM at idle.
That’s from their own profiling
on iOS,
where they were capped
at 15 megabytes total
and sat around 13.
On Android,
it runs a bit higher,
but still in that all-perk,
well under 20 megabytes
steady state.
For a dedicated voice recorder,
that’s plenty,
especially since
the Rabbit R1
has around 4 gigabytes
of RAM
even after the Android
conversion eats some of it.
For authentication
without a Google account,
the app has a sign-in
with other option
that supports
any SSO provider,
or you can use
an auth key
from the Tailscale
admin console
for headless
or kiosk-style setup.
There’s also a QR code flow
where the device
displays a code
and you approve it
from your admin console
on another machine.
One thing to keep
in mind,
though,
Tailscale only gives you
the network tunnel.
For actually pushing files
to your home computer,
you’ll still want something
like an SFTP server,
an SMB share,
or a simple upload endpoint
running on the receiving end.
Tailscale just makes
that endpoint reachable
as if it were
on your local network.
Want me to dig into
any of those
file transfer options?
No, no.
What I want to do
is I want to understand
better
exactly
how I register
the Tailscale app
on the Android.
So I can install it.
The server app
on the PC
has a USB-C cord
connected to the
Rabbit.
So I’m assuming
I can install Tailscale
using that USB cord.
Can I also authenticate it
at the same time
I install it
if I have the servers
already set up
with Tailscale?
Yes.
You can do both
over that USB-C connection.
Since your PC
already has Tailscale running,
here’s the cleanest path.
First,
on the Rabbit,
enable developer options
and turn on USB debugging.
Plug it into the PC,
then from a terminal
on the PC,
run ADB devices
to confirm it shows up.
Next,
grab the official APK
from PKG.
I know.
I think I know
how to do all that.
I just wanted to understand.
So Tailscale
looks like it’s
a better approach
than using
a Cloudflare tunnel.
It’s going to be
more secure.
Yeah.
For your setup,
Tailscale
is the better fit.
A Cloudflare tunnel
works by punching
a hole through
Cloudflare’s edge network
to expose a service
on your home machine,
which means that machine
has to run a tunnel daemon
and you’re relying
on Cloudflare’s infrastructure
as a middleman.
Tailscale instead
builds a private mesh
directly between your devices
using WireGuard,
so the connection
is end-to-end encrypted
with no third party
in the path,
and your home PC
doesn’t need to expose
anything at all.
For a dedicated recorder
just pushing files
to one specific machine,
that direct peer-to-peer approach
is simpler,
lighter,
and more secure.
Can I have multiple servers
on my computer
and each of them
have a different
Tailscale endpoint?
Yes.
Each instance just needs
its own state directory
and socket path
so they don’t collide.
On Linux,
you’d launch separate
Tailscale processes
with different
hyphen state directory
and hyphen socket flags,
and each one registers
as its own known
with its own Tailscale IP.
You can tag them differently
or put them on
different tailnets entirely.
On Windows,
it’s a bit more awkward
since there’s no native
multi-instance support,
but you can run them
in separate containers
or WSL instances
to get the same effect.
Can I serve up,
is there any kind
of server,
can I create
a tiny little server
and serve up
a web page
from within my desktop
computer using Tailscale?
Absolutely.
Tailscale has a built-in feature
called Serve
that does exactly that.
You run a tiny web server
on your desktop,
something like a Python HTTP server
or Nginx,
then point Tailscale Serve
at it with a command like
Tailscale Serve 8080.
That makes the page available
at your machine’s
Tailscale IP
or magic DNS name,
but only to devices
on your Tailnet.
So your rabbit,
your phone,
anything connected
to your Tailscale network
can hit it
and nothing outside can.
If you ever wanted it public,
you’d use Funnel instead.
But for your use case,
Serve is the right call.
Hey, Grok,
let’s talk about
a video,
it’s kind of a sketch
drawing technique
you see in videos.
I’m trying to think
of the company that does it,
who created it,
but the whole idea
was that,
you know,
you see a hand
at a whiteboard
and it starts drawing something
and there’s a narrator
in the background
doing something.
What is that called?
That’s usually called
a whiteboard animation,
also known as
a sketch animation
or doodle video.
The big names
in that space
are Video Scribe,
Doodly,
and Common Craft.
Common Craft
basically pioneered
the explainer style
with simple hand-drawn visuals
and a calm narrator.
RSA Animate
took it further
with more artistic,
fast-paced
whiteboard sketches.
Any of those ring a bell?
Yes,
RSA Animate
does a really good job.
So,
let’s say I want to create
a video
and I want it to be
in the style
of RSA Animate.
I don’t really even need
the hand
and the pencil
in the picture.
I just want to see it
being drawn
on a whiteboard.
You know,
how would I go about
creating a video
like that with AI?
Would I need to create
a custom tool first
or are there any custom tools
out there that do that?
Let me check
what’s out there right now.
Good news,
you don’t need
to build anything custom.
There are several
solid options right now.
The closest match
to what you’re describing
is Ancon AI.
You give it a topic
or script,
it writes the narration,
sketches each scene
stroke by stroke
on a clean whiteboard
and syncs the voiceover.
It has a no-hand mode
so you get just
the drawing effect
without a visible hand.
does something similar
but starts from images
you already have
and animates them
with an outline-first,
no-hand option.
Are these open source
or are these
subscription-based?
Mostly subscription-based.
Ancon AI,
Colpo,
and Animaker
all run on monthly plans.
Okay,
I want to make my own.
Okay,
so that’s first.
Or use open source,
you know,
open source parts,
using open source parts
but I definitely
want to make my own.
And I’ve got,
you know,
access to Claude
and Codex
and that can use
the frontier AIs
to drive this thing
but I want,
this is not
a commercial product,
this is a product
for me to create
my own,
my own talking,
you know,
my own videos.
I have a YouTube account
so I can just,
you know,
put them on there.
So,
that being said,
that’s,
how would I go about,
you know,
what are some ways
I go about
creating something like that?
Let me pull up
the open source options.
There’s a really
clean path here
and the good news is
you don’t have
to start from zero.
The best open source
render engine out there
is called
Whiteboard-Animator
on GitHub.
MIT Licensed,
runs on CPU only,
no GPU or API keys
needed for the core rendering.
You feed it a clean
whiteboard-style image
and it traces every stroke,
writes text word by word,
fills shapes with brush strokes,
and paces it all
to a narration audio file.
That’s your drawing engine.
Then you wrap it
with an AI pipeline.
The architecture
looks like this.
First, Claude or Codex
takes your topic
and writes a scene-by-scene script.
Each scene is one idea,
roughly 12 to 15 words,
with a visual description.
Second,
you send each scene description
to an image model,
something like Flux or GPT image,
and ask it to draw a clean,
black ink on white illustration.
Third,
you run each image
through Whiteboard-Animator
with a matching narration clip.
Fourth,
FMPG stitches the scenes together
into one video.
For the narration,
you can use 11 labs
if you want a polished voice
or Edge TTS,
which is free
and needs no API key.
There’s also a project
called HandDraw-Skill
that does almost exactly
this end-to-end.
It uses SVG,
Remotion,
FMPG,
and free Edge TTS,
and it’s designed
to be driven
by a coding agent
like Claude Code or Codex.
You could fork that
as your starting point
and customize the visual style
to match RSA Animate
more closely.
The one thing
Whiteboard-Animator
doesn’t do
is generate the images for you.
It only animates
what you give it.
So the image generation step
is where you lean on
Claude or Codex
to craft good prompts
or call an image API directly.
Want me to dig deeper
into any one of those pieces?
Yeah,
I’d like you to try
to figure out
how they
understand
A,
how to
know where to put
the pencil to draw
based on
the script.
Meaning,
so if I have like
a picture of an apple
and an orange
and a pear
and I first start talking
and I start with a white screen
and I first start talking
about the apple,
does it draw,
how does it know
to draw the apple?
and B,
how does it even,
is it drawing
by just removing
a mask?
Is it drawing,
removing a mask,
creating a raster mask
and only exposing
the parts of the line
where its cursor
or the pen
is going over?
Or is it vectorizing
the whole thing
and then drawing it
using vector shapes,
you know,
basically using
the vector shapes
as paths?
Great questions.
Those are the two
hardest parts
of the whole thing.
Here’s the honest answer
on both.
On the first question,
it doesn’t actually
understand the script.
It’s not doing
semantic matching
between apple
in the narration
and the apple
in the image.
What it does
is spatial heuristics.
Containers get drawn
before their contents,
shapes before their labels,
text and reading order,
small dots attached
to whatever glyph
they belong to.
So if your apple,
orange,
and pear
are just three separate blobs
on a white background,
it’ll pick one
based on position,
probably left to right,
top to bottom.
Not because it knows
you’re talking
about the apple first.
You can override that
with a region plan,
which is a JSON file
where you explicitly list
each region
and its reveal order.
There’s also
an optional Gemini step
where you pass in the image
plus your narration text
and Gemini proposes
which regions to draw
and in what order.
But even then,
the final order
comes from that explicit plan,
not from real-time
speech understanding.
So the smart part
is really just
a one-time LLM pass
that annotates your image,
not the renderer itself.
On the second question,
it’s raster,
not vector.
No SVG,
no path tracing
in the vector sense.
Here’s what actually happens.
It scans the image
for connected blobs
of non-white pixels.
Anything darker
than about 240 gray
becomes a component.
A small text detector model
flags which blobs
are words
so those get written
left to right
instead of traced
as outlines.
Then for each component,
it computes a skeleton,
basically the centerline
of the stroke,
and starts revealing
from a real endpoint,
like the tip of a V.
Closed shapes
get a single traveling front
that walks around
the boundary.
Fills get an outline pass
first,
then either an angled sweep
or bristled brush strokes
depending on size.
And anything with junctions,
like an X or a grid,
gets decomposed
into sequential pen paths
so it doesn’t just bloom
outward from the center.
Each pixel gets assigned
a reveal timestamp.
As frames render,
finished pixels
get committed permanently,
and only the pixels
currently being drawn
get a fade blend.
Everything streams out
through FMFPEG,
so it’s really
a pixel-level mask reveal
guided by skeleton geometry,
not vector paths,
but not a dumb
left-to-right wipe either.
The skeleton tracing
is what makes it feel
like an actual hand drawing
rather than a curtain opening.
The trade-off
is it only works well
on clean ink-on-white
illustrations.
Photos,
gradients,
textured backgrounds,
those fall apart
because the connected
component detection
can’t separate
meaningful strokes
from noise.
Okay, I got that.
I got that.
So, let me ask,
I’m just brainstorming here.
What,
has anyone tried
to do this
using an AI video?
So, with AI video,
you could give it
a picture and say,
I want you,
I think,
maybe you can say this,
I want you to
progressively disclose this
in this order.
Draw the apple first,
and then shade it red,
and then draw the pear second,
and shade it,
you know,
kind of a golden yellow,
and then finally,
in the bottom right-hand corner,
I want you to zoom in
and draw the orange last,
and shade it orange.
Yes,
people have tried
exactly that,
and there’s even
a research project
built for it.
It’s called VideoSketcher.
An LLM takes your prompt,
breaks the subject
into semantic parts,
and decides
the drawing order,
like cake base first,
then frosting,
then candles.
That plan gets fed
to a video diffusion model
trained on artist SVGs,
where each stroke
animates along its path.
So the model literally
generates a video
of strokes
appearing progressively
on a blank canvas
in a meaningful order,
not just a random wipe.
The catch is,
it’s a research prototype,
not a polished product
you can just open and use.
And the control is semantic,
not pixel-precise.
You can’t say,
draw the apple,
then shade it red,
then zoom to the bottom right
for the orange.
You get a rough order of parts,
but the model decides
the exact strokes.
What creators are actually
doing in practice
is the start and end frame trick.
You generate a blank whiteboard
as frame one,
a whiteboard with just
the apple outline
as frame two,
then a version
with the apple shaded red,
then one with the pear added,
and so on.
You chain those as keyframes
through C-dance or VO,
and the model interpolates
the drawing motion
between each state.
Kling’s motion brush
helps too.
You paint over the region
you want to animate
and leave the rest static,
so only the apple moves
while everything else
holds still.
So the honest answer is,
the idea works,
but nobody’s nailed
fine-grained,
script-driven,
stroke-level control
with a pure video model yet.
The deterministic engines
like whiteboard animator
still win on precision.
The video models win
on organic,
hand-drawn feel.
They just can’t follow
a strict order.
I see.
So,
uh…
So,
the practical takeaway
is this.
If you want pixel-perfect
control over drawing order,
stick with a deterministic
renderer like
whiteboard-animator
and let an LLM
just plan the regions.
If you want that
organic,
hand-drawn feel
where the strokes look alive,
lean on a video model,
but accept you’ll be
chaining keyframes
and babysitting the order.
Most people building
this for real
end up doing a hybrid.
Generate the clean illustration
with an image model,
run it through a stroke renderer
for the drawing motion,
then use a video model
only for the camera moves
or shading transitions
where you don’t need
exact stroke control.
That gets you maybe
80% of the RSA animate look
without fighting the model
on every frame.
Want me to sketch out
what that hybrid pipeline
would actually look like in code?
No, no.
Here’s what I want you to do.
Let’s think about this
for a second.
Let’s think about
a complete RSA animate type
drawing,
whiteboard drawing.
And when you zoom out
at the end,
you see like,
I don’t know,
there’s probably 30 or 40
different drawings
on the whiteboard.
But you started
at the top left
and you kept drawing
and you moved
and you panned
and scanned
and went back,
you know,
zoomed in a little bit,
zoomed out a little bit.
but the end
is in way out.
So if you wanted
to create that type
of video,
then it seems to me
that what you’d want
to do is
you’d want to
create the video
at a super high resolution
first
and then add
the camera movements
based on the narration
second.
Does that make sense?
That makes total sense.
And it’s actually
how the real RSA animate
videos were built.
cognitive,
the studio behind
RSA animate
worked on a giant canvas,
sometimes several times
the final frame size
and the camera
was a separate layer
that moved around it.
So your instinct
is exactly right.
You render the full
whiteboard at say
4K or 8K
with every drawing
placed in its final position.
Then you define
a camera path,
a sequence of keyframes
with timestamps,
each one saying
at three seconds,
be zoomed
into the top left
at 200%.
At seven seconds,
pan right to the apple.
At 12 seconds,
zoom out to show everything.
That camera path
gets driven
by your narration timestamps
so the drawing
and the voice
stay locked together.
Tools like Remotion
handle this natively.
It’s basically
a React component
where you animate
where you animate
a transform
with a big image.
I know Remotion.
So how do I,
what tool would I use
to script
creating that
extremely large
frame video?
You generate it,
right,
a grid of tiles.
Each tile
gets generated
at normal resolution,
say 1024x1024,
with prompts
that describe
what’s in that region
and what bleeds
into the neighboring tiles
so the seams match up.
Then you composite them
with a small overlap
and a soft blend
at the edges.
And you’ve got
one giant canvas,
maybe 8K
or 16K total.
The trick
is the prompts.
You feed each tile generation
the context
of its neighbors,
so a drawing
that straddles
two tiles
doesn’t get cut
in half.
Wait, wait, wait, wait, wait, wait, wait.
So are you saying,
let me back up.
Are you saying
you create a finished,
the finished render first?
Is that what you’re saying?
Yes, exactly.
You generate
the entire finished whiteboard,
all 30 or 40 drawings,
fully inked and shaded,
sitting in their final positions,
as one static image first.
Nothing is animated yet.
Then in Remotion,
you load that single image
and animate
a virtual camera window
that pans,
zooms,
and reveals pieces of it
over time.
Synced to your narration.
The drawing motion itself
comes from the camera
revealing what’s already there,
not from generating
strokes on the fly.
That’s what gives you
that RSA animate feel
where the whole board
exists from the start
and the camera
just discovers it
piece by piece.