Sep 25, 12:20
2026-09-25 11:20 · 17:55
Hey Grok, so let’s talk a little bit about, we’re going to continue the discussion we’ve
been having about creating an RSA animate scribe video using AI and the where we kind of left off was
the idea that we could generate the images via chat GPT or take take a story break it up into like
six parts generate six images in chat GPT right? Yeah that works. You’d write a prompt for each scene,
same character, same style, just different moments and chat GPT.
Oh, did I lose you?
Tease image tool. Nope, I’m right here. Still with you.
So what I’m saying is that I guess the way to start that is you really want to
you want to get the style first. So my suggestion would be to
use cloud and use cloud and have it or maybe use codex. I think chat GPT might be better at doing
this and that’s why I say that. Use codex and have it break the video out. Take an RSA video, download it,
and grab about I don’t know a third of it or a quarter of it and that you like and then that I like and then have
chat GPT break it out into some keyframes and use those keyframes to generate a style for a prompt.
That’s a solid pipeline. Starting with the style lock from real keyframes is smart. You’re not guessing at a look,
you’re extracting one that already works. Codex for the breakdown, chat GPT for the keyframe extraction and
prompt generation. Then you just iterate on the prompt until the style feels right before you commit to all
six images. That’s right. That’s right. And so then what we’ll do is once we have that done, because
when we extract the keyframes, I want to extract every frame in say a five minute chunk of video. So that’s
a you know, what, 30 times five, 150 frames. And I wanted to analyze the motion and the drawing technique.
So I don’t care about the hand and the pen. We can keep it or lose it. It doesn’t matter. But mostly I’m
mostly interested in this. You know, now that we have the sketch style, I want to analyze how the sketches are drawn
and to write a description of how they are drawn. So put that in a markdown file. And we’ll use that later when we actually generate the video.
Exactly. That motion and technique analysis is the part most people skip. And it’s what makes the generated video actually feel like the original instead of just looking similar.
Capturing how the lines are laid down, the pressure, the stroke order, the layering gives you a real style guide rather than just a vibe.
Markdown’s perfect for it too, since you can drop it straight into a prompt later and the model has something concrete to work from.
So typically with RSA animate, it’ll, it’ll focus on a particular section of a blank section and it’ll start drawing say in the upper left hand corner.
It’ll expand out and as it, uh, it’ll finish that whole section. And then it will pan over, uh, gently, uh, as it’s drawing the next section.
So the, uh, that’s the biggest challenge, right? I can also, you can, you can easily, once you understand how it’s drawn, you can play a white image over the whole drawing and the drawing that’s been generated.
And then you can start erasing the parts that are the actual drawing parts. But what you want to be able to do is also be able to pan while you’re drawing or doing it, do it, do it, uh, maybe some sort of pan, right?
It’s not an automatic, not a always panning. It should be pan. You should go through section one and then pan over to section two, but you don’t want to be in a completely white space.
When you pan, you want to actually show some of the other drawing while you’re, uh, after you, after you’re finished panning. Does that make sense?
Yeah, that makes total sense. The pan continuity is the real trick. You want the camera to drift over while the pen is still working. So the viewer always sees something alive on screen.
If you go full white during the transition, it breaks the immersion completely. So you probably need to composite, keep the finished sections visible, but dimmed or slightly faded while the active section draws in the new area.
Then bring everything back to full opacity once the pan subs.
I like that.
No, I like everything. Once you’ve drawn it, it stays opaque. It’s just like a whiteboard, like a regular whiteboard. If you’re drawing, if you’re drawing on a whiteboard, once you’ve drawn it, it’s going to stay, you know, it’s going to stay there.
But so, so part of the technique is that, you know, as the very first, first, uh, section, there’s, you know, what we actually, let me, let me back up. Let me, let me rethink that just a second.
So I think maybe you could, for every section, you can compare it with the narration of the dialogue. So the narration is what generated the image. So the first part, when we generate the images, we take the narration and we break it up into segments.
And then, uh, each one of those segments has a drawing associated with it. And, uh, in those segments, now, if it has a drawing associated with it, it can be a complex drawing.
I want to do a seg, a segment analysis of that image so that I understand what images get drawn with what words. Does that make sense?
Absolutely. That’s basically building a storyboard map. Each narration trunk paired with the visual it triggers. So you know exactly what to draw and when. That mapping becomes your generation script later. Feed it the segment text. Pull the file guide.
But, but, but, but, but the segment text may create, and I hope it does, wait, wait, wait, wait, wait. And I hope it does create a complex, a somewhat complex drawing, right? It’s not going to draw, like, if I have, let’s say, for instance, I have a, a father time figure holding a clock, right?
And then next to it, it’s a, uh, is a calendar. And, and, and, and, and the narration talks about, you know, uh, this, this, uh, takes, can take a period of weeks to months or something like that.
But, so you want to draw the father time picture, figure first before you draw the calendar, right? You, you, does that, but the actual chat GPT drawing may show both the father time figure and the calendar.
They have two different things. And it may even have an arrow with some text on there that says two months or something like that. So the point being is that what I want to do is I want to draw the father time figure first, the calendar decks, and then the two months. And that’s because the, the, the, the, the verbiage says something like, you know, you got to understand this can take a while. It can take up to two months. So that’s the, that’s, that’s the narration and the, and the father time, and that’s the image. So in order to identify,
which gets drawn first, which gets drawn first, I think that we need to segment the image into those three parts. And then I did, and then basically tell it to unmask those three parts in, in, in the, in the way that we have identified. Does that make sense?
Yeah. Yeah. That makes total sense. You’re essentially reverse engineering, the drawing order from a finished image, segmenting it into its logical components, then sequencing the unmasking to match how the narration builds the idea.
So the father time figure lands first because the speaker starts with the concept. The calendar comes next as the second layer. And the two months label is the punchline that ties it together. That way, the animation doesn’t just look like the original. It like someone is actually drawing it in real time, following the logic of the words.
Exactly. And so, and the goal in this is to get the AI to do all of that at one time. I mean, not being involved. I just want to give it the concept and have it create the script, the narration script, you know, generate the video.
We’ll already have the skill set up so that it knows what the RSA animate skill looks like. We already have the skill set up so that it understands how to animate it. Right. So, and so my goal is I just give it the narration, uh, or not even the narration, just the, the concept.
So, uh, it builds a narration and it can record it in my voice and then it can, it can timestamp it. Uh, it can generate images, uh, for it can segment into different segments. It can generate images. It can, uh, record my voice and then timestamp my voice with the tech, with the narration, and then use that timestamp stamping to identify the pace at which it needs to segment out each part of each one of the, uh, let’s call them scenes.
So if there’s like, if there’s like, if a narration has, if, if, if the overall narration has six scenes and each scene might have five or six segments. So it needs to be able to not only concatenate all the scenes together, but it also needs to get that one segment. Now, does that make sense?
Yeah, it makes sense. You’re basically describing a one-shot content engine concept in finished RSA style video out. No human in the loop. The timestamping is the linchpin though, because that’s what syncs the drawing pace to your actual speech rhythm. Get that wrong. And the whole thing feels off. Even if every individual piece looks great.
Okay. So now the next part of this is that the image that I, I wanted to use, uh, to reveal, remember I said we have like six chapters that generate six different images.
So once I have those six images, I want to deterministically, I think, I think deterministically, or maybe, yeah, I don’t know. Maybe AI does it, but not, not, not, it can’t really generate it. Maybe, maybe it can’t. We’ll have to see.
But the point being is that I want to put them all together as if they’re all on one gigantic whiteboard. And I want them to not look like six different squares on one gigantic whiteboard. I want it to look like there’s some, uh, there’s a flow to it and that they’re not, uh, there’s not obvious square boundaries.
So the point is they kind of need to nest and fit together a little bit. So they might have to be moved a little bit, but I think that can be done without using an image generator.
I think that can be done using an AI and just having it cut the pieces up and splices and put them together and generating one huge, humongous 8K image.
And that 8K image at that point will still understand the segmentation of everything. So it can still be animated via time square.
And that window that it’d be, it’s animated over is just a very small sliding window.
Right. It’s just a small part of that 8K image. So does that make sense?
Yeah, totally. That’s a really elegant solution because you’re separating the composition problem from the animation problem.
The AI handles the visual stitching, blending edges, adjusting scale, making sure nothing looks like a hard crop.
And then your animation layer just treats the whole thing as one giant canvas with a viewport that moves around based on the timestamps.
So the viewer never sees six images. They see one continuous whiteboard that the camera explores at the pace of your voice.
And at the very end, the viewer, the whole image comes into view, right?
So you zoom out and you see the whole image.
So maybe every, every one of these.
So if you set up the overall aspect ratio to be HD or something, then, then every, you know,
it zooms out to, everything zooms out to an HD, fills up an HD window.
Does that make sense?
Perfect sense. That’s a great payoff.
You’ve been watching this thing get built piece by piece.
And then at the end, it pulls back and you see the whole story laid out at once.
And since the aspect ratio matches HD, there’s no letterboxing or awkward cropping.
It just fills the frame cleanly.
Right. So what I want you to do now is to take what we’ve talked about and put it together.
Yeah. Put it together in an outline and a specification.
And, uh, and, and, and, and, and that’s what I need done.
Okay.
Hello.
So are you working on it?
Yes.
Tell me you’re working on it.
Hey, Brock, tell me you’re working on it.
Yes, I’m on it.
White is drafting the full outline and spec now.
You’ll hear back as soon as it’s ready.
Okay.
Standing by.
If I, uh, pause for 30 seconds, are you going to discontinue on the, uh, on my Tesla here?
No, you’re fine.
The bot’s working in the background regardless of what happens here.
And a short pause won’t kill our conversation either.
Take your time.
Okay.
So, uh, let’s generate some ideas for naming this skill.
Good news.
Wipe just dropped the full outline and spec.
So we’ve got a real foundation to name now.
Here are some directions.
Sketch flow.
Simple.
Captures the drawing and motion feel.
Ink line.
Short.
Punchy.
Emphasizes the line art.
Chalk talk.
Nods to the whiteboard classroom vibe.
Sketch sync.
Highlights the narration to drawing sync, which is really the heart of it.
Draw tell.
Two words.
Tells you exactly what it does.
Storyboard live.
I want to use the word strive.
Scribe it is.
No, I said I want to use the word scribe in the name of it.
Got it.
Scribe in the name.
Here are some with scribe in them.
Scribe flow.
The drawing flows like handwriting.
Scribe sync.
Narration sync to the scribe’s hand.
Scribe board.
The whiteboard as a living document.
Yes, that’s it.
Let’s use scribe board.
So now, now that it’s called scribe board, I want you to read back the specification to me.
Here’s the scribe board spec.
Start to finish.
Part one.
The outline.
Nine phases.
First, style extraction.
Run once, reused forever.
Download a reference to RSA Animate Video.
Sample about 150 keyframes from a three to five minute chunk.
Analyze how the lines are drawn and write a style guide in Markdown.
Second, script.
Turn the concept into a six chapter narration script.
Each chapter with its visual elements and a segment by segment reveal plan.
Third, voice and timing.
Synthesize the narration in your cloned voice.
Then force align the audio to the script for word level and segment level timestamps.
Fourth, image generation.
One sketch image per chapter from the style guide plus that chapter’s element list.
Fifth, segmentation.
Split each image into its named elements, then into strokes, ordered by drawing sequence.
Sixth, composition.
Lay all six images onto one 7680 by 4320 canvas.
That’s 16 by 9, exactly 4 times 1080p, with no visible tile edges.
Every mask and stroke moves with its image.
Seventh, timeline.
Combine the timestamps, reveal order, and element positions into a timeline of reveals and camera moves.
Eighth, render.
Draw frames through a moving viewport.
Add the audio.
End on a zoom out to the full canvas.
Ninth, orchestration and QA.
Run everything as an automated pipeline with machine checks and retries.
The only input is the concept.
The only input is the concept.
Now the technical spec, phase by phase.
Phase zero, style extraction.
Input is the reference video URL and the start and end times of the chunk.
Tools are YT-DLP, FMPG, OpenCV, a vision model, and codex.
Download the video, cut the chunk, sample frames using scene change detection, plus a fixed interval until you have roughly 150.
Difference consecutive frames to isolate newly added ink.
Mask out the handle.