Oct 16, 13:30
A lunch conversation. I walk a friend through the new AI workflow I used to make a commercial for the roster. My friend’s questions are in italics.
Three commercials, three workflows
I was telling Veronica how far the new workflow has come:
- The Ring My Case commercial (the first one, with my buddy). I was still building all the skills, and it took a couple of weeks just to get set up.
- My last paid commercial. I got $6,000 from the client, and it took 30 hours, because that’s what the client got billed. The difference there was that it used real recorded footage for the storyline and real voices: the client’s real voices and their real interview footage. So I had to cut through all of that.
- The one I did yesterday. I started it after lunch, on my phone, because I was at lunch, just Clauding.
Step 1: Get the story right
Story Analyzer
From the phone I said: take this URL, throw it in the Story Circle, run the Story Analyzer, and give me the score. It scored it. Then: “How can we get this storyline to 100?” It went through and came back with a higher score, 90-something.
I looked at it and thought, okay, but that’s scoring it as a document. How does it know to score it as a script for a video?
Analyzing video, not only text
So I have a skill now that takes the QuickTime file and does two things:
- The obvious audio transcript.
- Because video is visual, a visual analyzer as well.
It analyzes the video both visually and by sound, and gives me a contact sheet. It doesn’t grab a frame every one or five seconds; it grabs the key frames, the significant action: a camera lens change, that kind of thing.
What if you’d started without the video, with just the script or transcript?
It would have missed nuances, because those are part of the story too, especially in a commercial. As a director, I’d always argue that video is visual by its very nature, and its strongest component is emotion.
I tell people this, even at IBM: if you’re going to create a video, understand that video by its very nature has an entertainment component. If I see it in video, I want to be entertained. Of course, in corporate America the reply is, “No, this ain’t Netflix.” I get it. But you’re putting it into a video, and that’s why a talking head will bore the crap out of you if you keep it on too long.
From concept to video script: the StoryBrand framework
Most of the time you don’t start with a video. You start with a script, a treatment or a story. How do you go from a concept to something that will work as a video script? That’s the problem I couldn’t get past.
That’s what I’m saying. When you really understand the why behind the structure, the why behind excite, disturb, assure, then the framework gives you exactly that: hooks to hang your stuff on. If you’re starting from scratch, you have a framework to lean on.
A good StoryBrand story starts with one rule: you have to designate who the hero is. If you don’t have a hero, you don’t have a star, and everybody always roots for the hero.
Example: a commercial for Suzy’s (where we’re eating; I come here once a week). Who’s the hero? Not Suzy. You and I, as the characters, are the hero. Suzy is the Yoda: the one who gives us, the hero, what we want, which is a good lunch, a great meal.
This commercial:
- The boss is the hero.
- Dave is not the hero, because Dave screwed up. He’s the comic relief. But he’s also the one who says to the boss, “The reason it doesn’t work is that we need to get professionals.” He’s the reason for “unique creatives with AI, plus AI.”
- The hired gun, the roster creative, comes in. She’s still not the hero. She’s the guide, the Yoda.
That matters because this is for the roster, and the roster’s intent is to make the customer the hero. You come to the roster, you win, not the roster. We’re Yoda; we’re here to help you succeed.
The story structure nails the hero, the guide, and the plan and the path. It gives you the key parts of a great story, no matter what. But we still had the problem that it doesn’t understand it’s a video. It doesn’t have the emotion, the camera, the gravitas, the timing, the beats.
Step 2: The art layer
The visuals are the art, and that’s where I came in. When it first did it, it gave me a dry delivery, like “Hey, the Rock.” I said no, I need this to be more human.
Is that a direction to the story?
No, that’s the art component. It gave me all the beats. Now the art came in: visually, I needed this to be cinematic.
Look and tone: the Apple commercial
I fed it the latest Apple commercial, the new one with the little baby iMac that looks like an old iMac from the '90s, and said: see this Apple commercial? See the way it’s lit? I wanted a particular look and a particular tone.
The subtext was that it starts cold and goes warm. That’s why the break room is cold and bluish, and by the end it’s warm. When the roster creative comes in, she’s warm and wearing warm colors. All that subtext is happening on the art side.
How I described it
Did you take the story into Microsoft Word and start typing?
No, this was me talking. I saved the whole beat for you. It’s a good case study, and a great one for a Sherpa.
The simple way I did it was two things:
- “Take a look at this Apple commercial.”
- “Take a look at my cinematic filmic prompt.” That prompt has a lot of the language in it. “This needs to have a cinematic feel.”
For the audio, I said I need the way humans talk. Sometimes it’s the arms and the eyes, and that’s why Dave is doing his kind of thing.
The blind test
I did a blind test with Veronica. First I showed her your version, and she said, “This is good.” I said yes, it’s good; yours wasn’t bad at all. It was just missing these elements. Then I showed her mine and she said, “Oh, I get it now.” She saw the emotion, the way the lines were delivered, the color, the visual look and feel, the angles, the way the camera moves.
Step 3: Voices
Daisy Bird, then Voiceify
I have an API for Daisy Bird, a similar app that I bought, with 100 voices in it. I used those first and they were robotic. They sounded good but not human, just a flat delivery: “Hey, Dave.”
Then I took it into a modified Chippify that I call Voiceify, and ran it through a number of times: it needs to be more human, I need nuances, and so on. Now I can take any Daisy Bird voice, run it through Voiceify, and get back these awesome voiceified voices. (I’ll make sure I send it to you.)
Chippify, the foundation
For Chippify, I fed it a bunch of my own YouTube videos I’ve made over the years. I wanted the cadence, the spacing, the timing, the language I would use, so I could take any text script, Chippify it, and generate another script that sounds more like how I’d say it, along with some understanding of timing. I think it worked because it had my original 2010–20 videos to go through and identify how I talk.
How Voiceify makes voices human
Voiceify took Chippify as a basis. Then, using me as a template, I had it study the way people naturally deliver their lines: what makes them human, what makes voices human. Building Voiceify was this kind of conversation, and there are usually little things.
The other thing it does is break the lines into pieces, so it can concentrate more easily. It’s like filmmaking. Say we’re shooting a scene here in this restaurant. In short filmmaking you don’t keep the camera on for 10 minutes; you shoot two minutes at a time and just pick up your line. Camera’s back here. “Action.” “So tell me, how did you do this?” “Cut.” Next, move the camera, we’re getting in tighter, deliver the lines again.
In the same way, Voiceify concentrates on one sentence at a time, or a paragraph at best, so it can really refine it. It still has the context of the previous and next sentences, so it has the context of the whole, and that’s why it can deliver more emotion. You can specify the emotion you want: this is a question, they’re scared, they’re screaming.
Chippify built the foundation, and then I added my own filmmaking stuff.
The production order
Did you do characters, props and locations?
For this one, no. I wanted this to be the Auto-Tune of commercials. Once I got the script, all I did was the characters, the voices, the look and feel from Apple, and then locked it down. It took my stuff and the Apple stuff, and that was the look I wanted, with the cold and the warm. I have all those files, and I’m going to build a case study.
The order, which all happens pretty fast:
- Script, locked down first.
- Music. I cut to music before voice.
- Voices.
- Sound effects.
- Characters (heads only).
- Storyboards and animatic.
- Video renders in LTX.
- Edit in DaVinci Resolve.
Separate audio tracks in DaVinci Resolve
When I did mine, I made one full audio track with all the beats and timing, rendered as a single file with the different voices but no sound effects or music. Then I made the video for each scene or shot. Is that how you did it?
Yes, but everything goes into DaVinci Resolve now, which I love. I can see everything: my music, all my individual audio tracks, my videos. As it plays I get a feel for it, and I’m tweaking. Then I tell it: I tweaked the audio, the music fade-out, use that. That’s the new baseline.
That matters because I like doing J-cuts. A J-cut is when the audio for the next scene starts while you still see this image.
Music
So you add the music after the whole video is put together?
No, I cut to the music. The music is all the emotion; the music is how I get the feel. Once I get the script, to be honest, I do music next, all the time.
On all my movies, while I’m writing the script I’m listening to music that inspires the music I want for the film, every time. I find it on my Spotify playlists, that kind of thing. Part of that probably comes from Tarantino, who has some of the best music in scenes because the guy’s a freaking encyclopedia. He also pays royalties, and you can’t do that. But now with Suno I can get the feel.
One thing I’d argue: take the sound out of a film. Is it still good? Then put the music in.
I don’t even notice the music. In your video I didn’t notice it. I’m going to go back and look at it now.
I’m the opposite; it’s the first thing I notice.
Suno, YuE and the lyric-removal skill
I use Suno and YuE. YuE is awesome, but most of the time it produces lyrics, so I have a skill that removes them. I’ll give you that skill.
I used the same skill on the Apple commercial, because I wanted to study it. It took out all the voices, and now I could listen to the Apple music alone. You hear the snap, this click, and then “there you go.” When you hear it, you understand. That’s in the case study.
Rhythm in editing
As an editor, that’s what you use, and my editing prompts know all about rhythm. Rhythm is one of the key parts of editing. It doesn’t just mean the rhythm of the sound; it goes back to the subtext. In a horror film you might start the rhythm slow, boom… boom… boom, then the monster gets closer and it speeds up, and when the monster’s on you it’s boom-boom-boom-boom. Jaws literally used that. Editors cut to those rhythms, and the same goes for a commercial or a corporate video. It’s a key part.
Changing the music late
The music went through a couple of iterations, right to the end. I used one piece up until the last pass. Then, and this is so cool, I have another skill that would take any… I took the Apple commercial. (My friend steered back to the voices here.)
Sound effects
SoundSnap and the new open library
Then I bring in the sound effects. I have Claude connected to my SoundSnap, which is a Hollywood library, Hollywood high quality. It searches, drops everything into my sound effects folder, and drops each effect into my timeline where it should be. Boom, boom, boom. Then I tweak it.
It also already does audio-engineer-level sound. When it drops an effect in, it knows what level to set, normalizes all the audio, does all the fun stuff, so by the time it’s in DaVinci everything is mastered.
Today I added a second source. That’s why I love my dailies: today they gave me an open API library of sound effects, 55.2 million audio files covering sound effects, ambient sound and music. My friend sent the same one to the whole team yesterday. SoundSnap stays the main one; the new library is secondary.
Reading the script like a Foley artist
How can you do sound effects, like keyboard typing, before you have video? Does the AI just work out the effects?
Yes. It knows it’s an office, New York, urban; it picks up on that. I’ve also taught it: read the script from the standpoint of a Hollywood Foley artist. When a Foley artist reads a script, what do they do? I did the same thing for cinematography and for voice. I’m bringing those roles into my prompting, so when it goes off and chooses sounds, it does it as “I’m a Foley artist in Hollywood. What would I think? How would I read this script?”
Characters
I didn’t build any prop or location sheets, and I didn’t do full character sheets either, only their heads. That’s what blew me away on this quick run; I really wanted it to be a quick-run test.
Where did the characters come from?
Nano Banana Pro. I described the people I wanted and really went with the first take, because they were good. Dave looks great. The boss looks like a boss. The big CEO looks like a CEO. And the creative was cool.
I want to see your character sheets and how you did them when we get to your house.
Character drift
Did you have any character drift?
Yes and no. Nano Banana Pro held the characters. I used start frames, and part of the prompting was “hold the character,” and it held.
Not too beautiful
For me, I don’t want people who look too beautiful. That’s the giveaway for AI. Almost every AI video has these gorgeous people, and you think, that doesn’t exist in real life.
It depends. I’d argue that commercial branding is real life with all the boring parts taken out. So if you’re casting, you look for beautiful ugly people. There’s a little bit of that.
There’s also typecasting. If you and I were actors, we’d fall into a typecast whether we wanted to or not. You can break the mold, but two old guys with gray hair aren’t the guys doing the thing in the bar. If we did, it would be a comedy, and then it would work. If I want to be comic, I take a character and displace them. That’s comedy. “I got my Tesla.” “Oh, you’re ugly, but I like your car.” You’re 70 years old, for crying out loud, trying to get your pickups. Maybe you walk into a nursing home: “Hey, baby, you want a ride?” That would be a good short film right there.
Storyboards and the animatic
Once I had the characters right, I had it build the storyboard. The script was already done and locked before the music and everything else.
I wanted to create an animatic. I had the music, the voiceovers and the character faces. Now: create the storyboards that hit all the beats.
Storyboards vs. first frames
I think of storyboards differently from first frames. A storyboard represents the shot. In a boardroom scene where we cut between people, each cutaway is a shot, and I make a frame that represents that shot rather than a starting frame. Or is it better to use a starting frame?
To me they’re in the same ballpark. What I’m after in a storyboard, in AI filmmaking, is high fidelity, as if a photographer walked onto the set and shot a photo while this was going on. That’s a storyboard today, and it really can be a first frame, though the camera may not start where that frame starts.
I didn’t prompt them as first frames. I just said, “Create the storyboards from the script,” and that’s what it did. When I prompted the video, they weren’t first frames but Omni references. If you compare them with the video, they don’t match.
Why the animatic matters
An animatic is basically the storyboards with the music and the sound, and sometimes I already bring in the sound effects. So I watch the animatic with the whole script before I launch a single video render.
Before AI, the cost to get there was real. Before I went to shoot, we’d get our animator, someone like Eric doing it by hand, to do all the drawings. Then we’d show the client, because we’d be going to pay and hire the crew and so on, so they had to sign off. “You didn’t want the camera to move here? You signed off on the animatic, right here.” That was a $4,000 shot.
But when LTX can render that fast, at some point you think maybe we don’t need to do that.
Rendering in LTX
These were rendered in LTX, on my own box. That’s what blew me away.
LTX vs. H3
Supposedly H3 is better. My friend did a Seedance comparison and thought H3 did a pretty good job, unlike some others. My friend also went back and had it rewritten, using Eric’s comments.
On render times: one render took about 15 minutes for under six seconds. LTX took about seven minutes (“six, seven”); I’ll pull it up to check. I’m using a weighted model, a faster one; they’ve compressed all those things.
I can do that with mine too, but I figured, let’s just use the best there is.
Prompting the shots
From the script, it had already been fed the cinematic language. Because I knew this would lean toward the Apple style, I said: look at my cinematic Bible, look at this Apple commercial, and now write the prompts.
I used to have prompts for different engines and models, but not for LTX; this was brand new. It knew it was prompting for LTX, and I said: look at the LTX model, look at my prompting guide for action, and stick to the character count for each prompt. Fewer characters is better because you’re efficient. So it wrote efficient prompts.
I also had it analyze the camera moves in the Apple videos. That’s where these came from:
- The slow push-in on Dave (“Who did this?”). That came from understanding the commercial and cinematic language.
- The wraparound with the boss. The camera moves around, starting on him and ending on Dave. That’s classic good camera work: you follow the action. When we analyzed yours, it was all on sticks: cut, cut, the camera never moves. That throws away the power of cinematic language. The story structure won’t do that for you; it just goes “big, big, big.” The art says follow the action: start on the boss and turn the camera in a smooth, gimbal kind of way.
- The roster creative’s close-up. She gets up, “No, AI,” and the camera goes right in her face, because she’s the guide representing the roster. She’s not the hero, but she’s the Yoda, and Yoda gets full screen. That’s importance.
Holding detail: the 4K reference
What Eric and I discovered: if you need to keep detail, take the first frame (or whatever reference LTX used; I have to look back at how it did it), upscale it to 4K, and feed that back in. Eric has logos on the shirts, and that’s how you hold details. You have to feed it a 4K image.
So before you render the video, you make the image in Nano Banana and upscale it to 4K? With what, Topaz?
Topaz. My friend has a local Topaz for images but not for video, because that’s pretty expensive. I have both, and I’ve got a pretty good pipeline for video now. I still need to put together a pipeline for images; I have a decent one, and I’ve been learning a lot about how to upscale video.
But all these video models downsample the images anyway. Why would a 4K image help?
In the tests we did, giving it a 4K image gives you a better result even after it’s downsampled.
Do you always upscale the whole video to 4K?
No. My 1080p upscales are really sharp, and at some point they’re too sharp. When you want a cinematic look, I also keep it at 24 frames, because that gives it the cinematic feel. Sometimes I hold back on purpose.
Re-rendering faces
My friend knows a way, on MiniMax, of going back and re-rendering faces in fast-moving shots so the model gets them right, because it tends to blur them with motion blur. The same goes for distance: someone standing over there may not have the same face when they’re far away. There’s a way to fix that.
Editing in DaVinci Resolve
When renders finish, they drop straight into DaVinci. It’s in DaVinci from pretty much the start: once I have the music and the audio, I have the animatic, because I need to see the feel of it right away. Then, while I watch in my editor, I tweak, pin it back, tweak.
Claude as the assistant editor
The music track is separate from the voice audio. If I need to cut the music, Claude’s a freaking mastermind. Once I lay things down and move stuff around, I just say: “I moved some stuff around. Go back and fix my audio; it’s on track A17.” Everything is properly labeled; I’ve set up the structure for that.
I tweak everything. I move sound effects around and say “go tweak all of them,” like talking to an engineer, and everything comes back in line.
If a line isn’t right, I say: “Clip number 17, redo that audio. I need him to sound more scared.” Boom, it fixes it and throws it back in, and I only had to touch that one little piece.
Ducking the music
For the final song I switched at the last minute and used two songs. I dropped them in and said: do all the ducking under the voiceover. It did, and it knows the levels from engineering standards. But the first pass was too stark, still too sharp, the way it ducked in and out. I said: soften that, make it gradual. It went through, changed every section and showed me all the numbers: it was at 0.5, now it’s at 0.875, for each section. I said okay, that’s good.
I always listen on headphones while I’m working, then listen again without them, because most people won’t have headphones.
That’s not how I do mine. I like what you’re saying much better.
J-cuts, L-cuts and “start late, leave early”
How do you know when to J-cut? There are like four kinds of cuts: J-cut, L-cut…
A J-cut is where you hear the sound from the next scene before the scene shows up. That’s the art of it. Every frame counts, and you’re trying to push the action.
I tell Eric this: he loves his images so much that he holds the cut too long. It’s only a second and a half, but cut a second off. You don’t need it. It’s not doing anything for the story; all you’re doing is saying “Oh, that’s pretty.” Holding it a second longer is good for you, but to the audience it’s a waste of time.
Same with the audio, and I’ll show you the J-cut. The way LTX rendered the boss, he has a whole lead-in, and the audience doesn’t need to see that. I call it “start late, leave early”: you start late in the scene, because all we need to know is that the dude’s frustrated. I started his audio right before the cut, underneath the previous clip, so when you see him you’ve already heard it, and the mind filled in the rest. That’s a J-cut. The audience picked it up immediately because they filled it in.
Clip lengths and keeping sync
My scripts are timed: from second one to three you do this, from three to six you do that. If you start sliding clips around, doesn’t everything get screwed up? Do you tell it how many seconds to render?
No. LTX renders the whole thing and puts it into DaVinci, and I cut there. When I have something re-rendered, I re-render the whole clip and replace it, and the cut stays where it was. I don’t change the timing of that cut.
If I have a two-second clip and I’m cutting the front off and leaving the back where it is, that’s fine. But the next clip was butted up against it, and the audio is synced to it. How do you match it?
I have to look at it in context, but take the scene where Dave puts his hand up (“Who’s doing this?”) and it cuts to the boss going “Oh my God, what did I just do?” LTX rendered the whole Dave clip, and Dave’s clip runs another second, but I cut off Dave early and brought in the boss, and I cut the boss clip early too. If you look at the full clips, one would overlap under the other, but I cut them anyway.
The voices don’t matter because they’re separate tracks. The music doesn’t need editing because it’s independent; the music is what drives everything.
But eventually the whole thing shrinks by a couple of seconds, and the music doesn’t.
Right, and that’s the tricky part at the end. I cut to a completely different track with a similar feel. Before, it was one long piece of music; at the end I changed it all and added a new layer, two completely different music pieces. One fades out when the boss is down on the phone: right before he hits, the music stops, because that’s the “crap, what am I going to do now?” moment. The music is completely gone. I made that decision on purpose.
The finished cut is a comp
So you rendered everything at 720p, hit the button to compile it all together, and then how do you upscale the whole thing?
That’s what I’m going to show you now. To me, what I made is really a comp. In the real world, I’d show this movie to the client: here’s a commercial I just did. It’s a great case study. But first we’d say: this is a good comp; this isn’t the real thing. “What do you mean? We did all that work.” Good job on the comp. Now we upscale everything. We go back to those key frames and upscale, upscale, upscale. We’re not changing the story or anything else, just raising all the visual quality.
Can’t you just take the whole video and upscale it?
You can, but you’ll lose some quality, because you need that 4K Omni reference to get a better image on the other end.
Do you know what latent upscaling is? Because it uses all those references, isn’t that already baked in?
That’s what I’m curious about, and I’m happy to test both. It’s just been my experience that if we start at the higher scale and then upscale, it’s better. Let’s test that out.
How Eric does it
Eric goes 4K all the way. He makes all his images 4K, because he knows that’s how you hold the details: all his characters, locations and props are 4K. He doesn’t render at 720; he renders at 1080 and upscales to 4K at the end, so everything is 4K.
Rendering at 1080 is expensive. LTX can’t render 1080, can it? I think H3 can, but only for about two or three seconds, so you can’t do much.
You can string them together; I’d have to look at that. But he renders at 1080 for sure. I don’t render at 1080 myself. The LTX output was about 940, 944 or something like that. I have it there; I’ll look at it.
Follow-ups
- Build the case study from this commercial (I saved the whole beat and all the files). It would also be a great one for a Sherpa.
- Send my friend Voiceify.
- Give my friend the lyric-removal skill.
- Show my friend the character heads and how I made them when we get to my house.
- Show the J-cut in the edit.
- Show the upscaling pipeline when we get back.
- Test upscaling the whole video (latent upscaling) against going back to 4K key frames.
- Look up the actual LTX render resolution and render times.
Asides during lunch
We ordered a Diet Coke, lunch specials with soup and egg rolls (beef broccoli with hot and sour soup, and shrimp delight with hot and sour soup), steamed white rice instead of fried, and a box to go.