A Director with an AI Film Crew

(For the TLDR crowd: ComfyUI, LTX 2.3 are good, Fanxiang SSDs are less good, and Claude can sync lipsync clips to a song and color-grade more easily than a human can. You will however end up sending it a lot of memos to refine its editing.)

From Wizard’s Car Chase to Pride & Planetoids

Just about a year ago, I made my first music video, Wizard’s Car Chase. It started with a Midjourney image of a modern Wizard’s Battle in the style of the Hildebrandt Brothers. Then I generated a bunch of random videos in Midjourney out of curiosity. Then I saw a writer acquaintance doing music videos for one of her worlds (human lyrics, Suno songs, Midjourney footage) and I wanted to try something similar. A few attempts in Suno later, I had a song, and about six-seven hours worth of editing across two days got me a video.

Then came the releases of Dragon’s Teeth and Pride & Planetoids. For the former, I did a series of trailers and vignettes covering the whole series to date. It was just Suno music and Midjourney video footage, held together with editing and silent movie intertitles. For the latter, I only had one book to cover, but I used much the same process with just a few wrinkles. I learned how to use Google’s Nano Banana for character reference sheets and for key framing. I also learned how to do first-frame last-frame animations in Midjourney, instead of letting it do its own thing. But there was one Planetoids song Suno had made for me that I really liked. I didn’t know what to do with it.

A Music Video Where the Characters Sing

Then old computer died, new computer came. I decided to try making a music video where the characters actually sang. I got some footage out of ComfyUI, using LTX 2.3. Things like Darcy and unnamed crew members singing, the Last Repose traveling through space and firing up its main gun. Then I combined that with existing footage generated for this video and this video. I messed up the initial lipsyncing so badly that I asked Claude if it could create a fresh Shotcut .mlt project with just the song track and the singing clips in place and locked down. It could. I started over, but still couldn’t get the edit to work until I brought in some reaction footage of Elizabeth to flesh it out.

The video was now mostly done, but the catch was that the interior footage didn’t match very well. The Midjourney stuff was very bright and saturated, while the LTX stuff tended more brownish. I asked Claude if it could color the video to match one clip of Elizabeth. Claude tried. It took a couple of passes to get the mlt file to a color grading I could live with, but it was a lot less trouble than trying to do it myself. Some five to eight months after I’d initially gotten the song from Suno, the last video for Pride & Planetoids was done and uploaded. (Look for it on YouTube, on November 30!)

Enter Phantom Queen

Meanwhile, I’d started working with Claude on a trailer for Phantom Queen, the prequel to the Hunter Healer King books. Claude proposed a fairly workable shot list for it based on a scene I wanted. I did end up simplifying some shots. And if not for a minor computer catastrophe (more about that later), I would probably have ended up adding some character footage that was not in the original shot list.

Giving the Characters Voices

I wanted actual lines of dialogue with some degree of emotion. I ended up choosing DramaBox TTS, built on the audio branch of the LTX 2.3 model. Once I downloaded it to a folder and gave Claude access, I would tell Claude what characters I wanted. It wrote (and rewrote as necessary) a .bat file and some commands to run to generate audio. This was a long, tedious process with many “takes” that were either bad emotionally or bad audio quality or just off for the character.

Although, in all fairness, nobody held a gun to my head and made me develop voices for Maxim, old and young Jerome, Ambrose, and Victor. Didn’t need them for the trailer. It just seemed smart to do Urban’s entire family while I was at it. Lines for Mara the heroine, the tyrannical Queen and a minor supporting character were also recorded.

The main lessons I learned from this process were that I needed a minimum of 10 seconds to be able to clone a randomly generated voice that I liked, and that DramaBox defaults to American pronunciation for short sentences. In order to get a suitable British accent for some of Urban of Storm’s lines in the trailer, I had Claude pad them out with a few extra sentences in front of the sentences I actually needed.

Generating Footage, and the Drive That Died

Then on to generating footage for the Phantom Queen trailer. I discovered the Batch Count feature in ComfyUI (next to the “Run”) button, which meant that I could generate multiple takes of whatever I was prompting for, generated one batch at a time. Then I discovered you could start one job of however many batches, change the reference images and the prompt, and start another job to run after it. Amazing!

And then…the six month old Fanxiang S501Q hard drive in my new computer decided it was time to start degrading. The first thing it ate was one of the LTX models I was using. I ordered a replacement drive (Western Digital, although from their Green line, so not the highest end). While waiting for it to arrive, I decided to finish the trailer with only the footage I had, and some music clips I’d used for earlier Hunter Healer King videos.

Editing with Claude

Claude made a decent rough cut with the dialogue in the correct places. (Keep in mind, it had access to the original shot list). I slowed down and repositioned the more sinister piece of music I’d brought into the project folder, adjusted a crossfade, and gave Claude feedback on some other transitions. We kept trying different things with the later part of the trailer, which was where I was short on footage. Finally, I was mostly satisfied with that part.

I showed Claude one of the existing promo clips for the series, so it could see what the outro screens looked like. After a bit of confusion about what the font I was using was called, Claude managed to mimic that. I showed the trailer to some Discord acquaintances, and discovered that the Queen wasn’t reading as sinister enough. I dropped her line of dialogue, and had Claude use sound effects and color grading to make her scene more menacing.

Here’s the kind of memo I mean, sent after Claude’s first pass at that scene:

Okay, I felt like that was maybe too much slow motion on the queen. It looked like the glow switched back and forth between purple and amber a couple of times and not in a way that really worked. Pushin either didn’t work or was too subtle to register, so please don’t bother with it on your next pass.

The final sequence of Urban’s giant Drill breaking through to the underground city looked overly bright, like a 1990s TV movie. I got Claude to redo the color grading to make that part darker and more dramatic, which also took a couple of passes. All in all, it took fourteen versions made in around 5 hours to achieve the finished cut. Would have gone faster if I hadn’t had to do other things while Claude was working and came back to it quicker. It took less out of me than manual editing would have, but still reflected my vision of what the trailer should be.

Off to Surgery

The replacement drive arrived this morning, so I’m off to spend Halloween afternoon performing brain surgery on my new computer. I wrote this post while I waited for it, and I’m sending it out the way a Victorian mad scientist leaves a diary behind: here is what I learned, in case the next step goes badly. If you’ve survived a drive failure yourself, I’d like to hear what you wish you’d backed up. In the meantime, the book all this footage was made for, Phantom Queen, is up for preorder now, and the trailer goes live on YouTube just after the release.