Video - https://youtu.be/zldbJpgUBqc

Overview

ComfyUI has native support for both SAM3 and FILM/RIFE frame interpolation, so I figured why not add those in to my LTX 2.3 workflow as well? There are also more and more LTX LoRAs available on huggingface, so I've been playing with those as well.

On top of that, I've updated some of the control groups and included a few more options for handling plain video (not pose/canny) input.

This workflow does everything the previous version did - and more! This means it still has a great many features, including:

VRAM Saver

Text to video

Image to Video

Custom Audio (lip-sync)

Union Control Net

LoRAs

First Image / Last Image

Best Face Swap

Workflow Information

Compared to last week there are loads of changes, which I've detailed below. See previous posts for more information on older groups such as the video extender.

Custom Node Requirements

One thing which hasn't changed since the previous release is the custom node requirement, meaning you have nothing new to install if you've already been using that. I do try to keep the custom node requirements to a minimum as I think it makes life easier in general.

If this is your first time, then you'll need to install those listed in the workflow note:

Model File Downloads

There are lots of options for LTX and you can chose which model files you'd like to download and use. I mostly use the files converted by Kijai...

... which is also what the workflow defaults to using:

If you prefer GGUF files, then you should instead connect the GGUF load to the Init group:

Or you can use the checkpoint versions:

Extra Model Downloads

Compared to last week, the extra models needed are for SAM3 and your choice of frame interpolation model:

I quite like FILM, but RIFE is a good choice too.

You can now more easily use a wide variety of the available LoRAs - such as those for inpainting, googly eyes and more! I've put a few suggestions in a workflow note for LoRAs which have either LTX or Apache licences:

Main Changes

The main changes are detailed in the "updates" note:

Frame Interpolation

The models may be old, but native support for them in ComfyUI is new - and now you can use FILM/RIFE in the Frame Interpolation group:

Take 24fps videos and make them into silky smooth 48fps videos in mere moments! Far better than the LTX Stage 3 FPS upscale in my opinion.

If you chose to use video extend, then this group will interpolate the extended video. If not, then the interpolation will apply to the standard video generation.

APG

In the Experimental Model Alterations group, you can now try APG (Adaptive Projected Guidance):

This node works best when you're not using a CFG of 1, but even with the distilled models it does make some changes to the video output. The default values are fairly reasonable, but experiment away to see what happens! Personally I prefer APG generations, as I think it helps LTX follow the prompt better, but the choice is yours.

One Video To Rule Them All

There is now just a single group for selecting your input video:

This gets used for all Input Video tasks such as Union ControlNet, Video Inpainting etc

As such, the Video Control panel has also been updated, including more detailed notes:

It is now as easy as 1, 2, 3 - and remember - using First Frame / Last frame is a totally different mode to using a ControlNet!

When using the Plain Video Guide "ControlNet" you can opt to disable the Stage 2 Processing:

This will allow Stage 2 Upscaling to change the video more, and hence follow your prompting. This can be combined with changing the options for the Stage 2 Alternative Sigmas, such as lowering the denoise for slightly less change to the original input video:

Also when using the Plain Video Guide "ControlNet" you can now opt to instead use an Empty Latent instead of the encoding the video itself. This allows for changes to things such as the aspect ratio, and to allow prompting to impact the video more heavily:

When using the Plain Video options, you can also opt to enable masking:

... then you can easily use the various inpainting LoRAs. (N.b. I've only used the "Outpainting" LoRA for inpainting so far - no video cropping/padding groups as yet!)

The masking has various options which you may wish to change such as the mask colour, expansion and blockyness:

Different LoRAs use different colours, thus if using the "outpainting" LoRA from the notes for inpainting, then you'll want to use a black mask instead:

Note that we're now getting some differences with the types of LoRA coming from the community and those from Lightricks. Some LoRA files need the IC-LoRA Guide node to be used, vs the older AddGuide (or MultiGuide) nodes. If the model card on Huggingface shows it is from Lightricks, or says it needs IC-LoRA Guide, then chances are you should use the IC-LoRA Guide ControlNet choice:

So far the only one that seems to need that option is the "outpainting" LoRA, but others may need it as well.

You can still load the actual LoRA itself using the normal LoRA Loader:

I've not included any video cropping nodes, but you can manually created cropped videos in your favourite video editor if you'd like to do outpainting tasks, or add the Pad Image For Outpainting node.

For the time being, at least, it is still only the Union ControlNet LoRA file which needs a scale reference value. Hardly complicated all, I know!

I'm having a blast with the various LoRAs listed in the note, such as the "googlyeyes" one:

I've also fixed a few small issues I'd not spotted before with some pre-processing for stage 2 upscaling, so things like Best Face Swap and Union Control now also work better than before if you're upscaling!

Enjoy :D