Home Creators Posts Import Register Favorites Logout

Content

Wan 2.1 text-to-video and image-to-video in SwarmUI with the CausVid LoRA at just 8 steps; on an RTX 5090 a clip took 90 seconds versus 170 on an RTX 3090 Ti.

Uses ComfyUI as the SwarmUI backend with Sage Attention, ready T2V and I2V presets, RIFE frame interpolation and GGUF models for low VRAM. Includes the SwarmUI installer and model downloader and the 1-click ComfyUI installer.

Tutorial Link

https://youtu.be/XNcn845UXdw

Tutorial Info

Wan 2.1 is still the very best local AI video generation model and it just became even more amazing and faster with CausVid LoRA. Now with utilizing ComfyUI backend power inside SwarmUI and my automatic installers to utilize Sage Attention, you can very fast generate very high quality AI videos with Wan 2.1 and CausVid LoRA at just 8 steps.

Tutorial Links

VIDEO CHAPTERS

  • 0:00 Introduction & Amazing Demo

  • 0:23 Tutorial Goals: Video Gen (1.2.1), Speedups (CowsWith, Rife)

  • 0:35 SwarmUI: Installation & Update Process

  • 0:57 SwarmUI Downloader: 1.2.1 Model & CowsWith Lora

  • 1:48 Optional: Integrating Models with ComfyUI

  • 2:12 SwarmUI Start, Config & Rife Interpolation

  • 2:38 Image-to-Video: Importing Presets

  • 2:51 Image-to-Video: GGUF Model Selection & VRAM

  • 3:09 Image-to-Video: Optimal Resolution & Aspect Ratio

  • 3:21 Image-to-Video: CowsWith Lora & "Fast CowsWith" Preset

  • 3:44 Image-to-Video: Key Parameters (Steps, CFG, Init Image)

  • 4:04 Image-to-Video: Creativity (0) & Frame Count

  • 4:26 VRAM Management: Avoiding Shared VRAM Slowdowns

  • 4:40 Image-to-Video: Rife x2 (Double FPS) & Advanced Settings

  • 4:56 Image-to-Video: Trimming Frames & Crafting Prompt

  • 5:24 Dual GPU I2V Gen: RTX 5090 vs 3090Ti

  • 5:55 I2V Speed, VRAM & First Result Analysis (RTX 5090: 5.7s/it)

  • 6:35 First I2V Result Review & Iteration Needs

  • 6:52 Second I2V Result: AI Fixes Missing Parts!

  • 7:20 Recap: Power of Optimized SwarmUI

  • 7:33 Text-to-Video: Switching & Model Setup

  • 8:04 Text-to-Video: Applying T2V Presets

  • 8:19 Text-to-Video: Key Parameter Differences

  • 8:34 Text-to-Video: Setting Resolution & Rife

  • 9:02 Text-to-Video: Prompting & Ensuring Lora

  • 9:17 Speed vs Quality: T-Cash & Sage Attention

  • 9:30 Text-to-Video: Dual GPU Generation Start & Setup

  • 10:03 Text-to-Video: VRAM Check & Speed Expectations

  • 11:51 Text-to-Video Speed Analysis: 5090 (8.4s/it) vs 3090Ti (18.2s/it)

  • 12:01 Text-to-Video Result (576x1008) Review: "Really Great!"

  • 12:55 Teaser: "My Diffusion Based Upscaler" & Quick Peek

  • 13:10 Upscaler Features: Splitting, Per-Clip Prompting/Upscaling

  • 13:35 Upscaler Features: Auto-Caption (CogVLM2), Ratio Control

  • 13:45 Upscaler Features: Batch Processing & Max Frame Control

  • 13:51 Upscaler: Quality Goal (10x+), Optimizations & Ideas

  • 18:01 Upscaler: FFmpeg Presets, Dev Status & Vision

  • 18:13 Final Recap: Hope You Enjoyed & Generated Videos Review

  • 18:20 Generated Video Quality & Time Assessment: "Magnificent!"

  • 18:24 Final Timings: RTX 3090Ti (170s) vs RTX 5090 (90s)

  • 18:32 Your Choice: Resolution, Frames, Speed & VRAM Balance

ABSTRACT

Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. The generation of a single frame requires the model to process the entire sequence, including the future. We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly. To further reduce latency, we extend distribution matching distillation (DMD) to videos, distilling 50-step diffusion model into a 4-step generator. To enable stable and high-quality distillation, we introduce a student initialization scheme based on teacher's ODE trajectories, as well as an asymmetric distillation strategy that supervises a causal student model with a bidirectional teacher. This approach effectively mitigates error accumulation in autoregressive generation, allowing long-duration video synthesis despite training on short clips. Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models. It enables fast streaming generation of high-quality videos at 9.4 FPS on a single GPU thanks to KV caching.

Related posts

All posts in this topic: AI Video Generation and Animation Apps · New to SECourses? Start Here: Best and Most Popular SECourses Posts

Files

Wan 2.1 Text-to-Video T2V & Image-to-Video I2V Tutorial for SwarmUI with CausVid LoRA Extreme Speed

Wan 2.1 is still the very best local AI video generation model and it just became even more amazing and faster with CausVid LoRA. Now with utilizing ComfyUI backend power inside SwarmUI and my automatic installers to utilize Sage Attention, you can very fast generate very high quality AI videos with Wan 2.1 and CausVid LoRA at just 8 steps. 🔗Follow below link to download the zip file that contains SwarmUI installer and AI models downloader Gradio App - the one used in the tutorial ⤵️ ▶️ https://www.patreon.com/posts/SwarmUI-Installer-AI-Videos-Downloader-114517862 ▶️ How to install SwarmUI main tutorial : https://youtu.be/fTzlQ0tjxj0 🔗Follow below link to download the zip file that contains ComfyUI 1-click installer that has all the Flash Attention, Sage Attention, xFormers, Triton, DeepSpeed, RTX 5000 series support ⤵️ ▶️ https://www.patreon.com/posts/Advanced-ComfyUI-1-Click-Installer-105023709 🔗 Python, Git, CUDA, C++, FFMPEG, MSVC installation tutorial - needed for ComfyUI ⤵️ ▶️ https://youtu.be/DrhUHnYfwC0 🔗 SECourses Official Discord 10500+ Members ⤵️ ▶️ https://discord.com/servers/software-engineering-courses-secourses-772774097734074388 🔗 Stable Diffusion, FLUX, Generative AI Tutorials and Resources GitHub ⤵️ ▶️ https://github.com/FurkanGozukara/Stable-Diffusion 🔗 SECourses Official Reddit - Stay Subscribed To Learn All The News and More ⤵️ ▶️ https://www.reddit.com/r/SECourses/ ▶️ CausVid LoRA Official repo : https://github.com/tianweiy/CausVid VIDEO CHAPTERS 0:00 Introduction & Amazing Demo 0:23 Tutorial Goals: Video Gen (1.2.1), Speedups (CowsWith, Rife) 0:35 SwarmUI: Installation & Update Process 0:57 SwarmUI Downloader: 1.2.1 Model & CowsWith Lora 1:48 Optional: Integrating Models with ComfyUI 2:12 SwarmUI Start, Config & Rife Interpolation 2:38 Image-to-Video: Importing Presets 2:51 Image-to-Video: GGUF Model Selection & VRAM 3:09 Image-to-Video: Optimal Resolution & Aspect Ratio 3:21 Image-to-Video: CowsWith Lora & "Fast CowsWith" Preset 3:44 Image-to-Video: Key Parameters (Steps, CFG, Init Image) 4:04 Image-to-Video: Creativity (0) & Frame Count 4:26 VRAM Management: Avoiding Shared VRAM Slowdowns 4:40 Image-to-Video: Rife x2 (Double FPS) & Advanced Settings 4:56 Image-to-Video: Trimming Frames & Crafting Prompt 5:24 Dual GPU I2V Gen: RTX 5090 vs 3090Ti 5:55 I2V Speed, VRAM & First Result Analysis (RTX 5090: 5.7s/it) 6:35 First I2V Result Review & Iteration Needs 6:52 Second I2V Result: AI Fixes Missing Parts! 7:20 Recap: Power of Optimized SwarmUI 7:33 Text-to-Video: Switching & Model Setup 8:04 Text-to-Video: Applying T2V Presets 8:19 Text-to-Video: Key Parameter Differences 8:34 Text-to-Video: Setting Resolution & Rife 9:02 Text-to-Video: Prompting & Ensuring Lora 9:17 Speed vs Quality: T-Cash & Sage Attention 9:30 Text-to-Video: Dual GPU Generation Start & Setup 10:03 Text-to-Video: VRAM Check & Speed Expectations 11:51 Text-to-Video Speed Analysis: 5090 (8.4s/it) vs 3090Ti (18.2s/it) 12:01 Text-to-Video Result (576x1008) Review: "Really Great!" 12:55 Teaser: "My Diffusion Based Upscaler" & Quick Peek 13:10 Upscaler Features: Splitting, Per-Clip Prompting/Upscaling 13:35 Upscaler Features: Auto-Caption (CogVLM2), Ratio Control 13:45 Upscaler Features: Batch Processing & Max Frame Control 13:51 Upscaler: Quality Goal (10x+), Optimizations & Ideas 18:01 Upscaler: FFmpeg Presets, Dev Status & Vision 18:13 Final Recap: Hope You Enjoyed & Generated Videos Review 18:20 Generated Video Quality & Time Assessment: "Magnificent!" 18:24 Final Timings: RTX 3090Ti (170s) vs RTX 5090 (90s) 18:32 Your Choice: Resolution, Frames, Speed & VRAM Balance ABSTRACT Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. The generation of a single frame requires the model to process the entire sequence, including the future. We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly. To further reduce latency, we extend distribution matching distillation (DMD) to videos, distilling 50-step diffusion model into a 4-step generator. To enable stable and high-quality distillation, we introduce a student initialization scheme based on teacher's ODE trajectories, as well as an asymmetric distillation strategy that supervises a causal student model with a bidirectional teacher. This approach effectively mitigates error accumulation in autoregressive generation, allowing long-duration video synthesis despite training on short clips. Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models. It enables fast streaming generation of high-quality videos at 9.4 FPS on a single GPU thanks to KV caching. Music provided by NoCopyrightSounds : Music Credits : https://gist.github.com/FurkanGozukara/8e16fcf8e82b68e71352e41cdc0ea395

Comments

Hockey

65 s/it with RTX 3080 16GB Laptop GPU (img2video), all on VRAM 13.8 GB. Settings: 1024x1024 input image Video output: 640X640 Steps: 8 CFG Scale: 1 Sampler: UniPC Init Image Creativity: 0 Init Image Recomposite Mask: V Model: Wan21_14b_Image_to_Video_720p_GGUF_Q6 Video Frames: 98 Video Steps: 8 Video2Video Creativity: 1 Video Format: h265-mp4 Video Frame Interpolation Method: RIFE Video Frame Interpolation Multiplier: 2 Video FPS: 16 Video Min CFG: 1 Lora: Mixed/Wan21_CausVid_14B_T2V_lora_rank32 Maybe the laptop version of RTX3080 is really slow? Output 6 seconds long, 1.57 MB, 2191 kbit/s

Klaus The.

Freshly installed the swarmui and downloaded the WAN modelspack. Imported your presets and loaded model, lora and applied your preset. Sadly, can't generate videos, there always comes the error "Invalid value for parameter Init Image". I chose an image, clicked the "Use closest aspect ratio" but always get the following error, tested in Firefox and Vivaldi(Chromeengine):

Klaus The.

[Error] Error in websocket handler: System.AggregateException: One or more errors occurred. (Invalid value for parameter Init Image) ---> System.Exception: Invalid value for parameter Init Image ---> System.ArgumentException: Data is empty! (Parameter 'data') at SwarmUI.Utils.Image..ctor(Byte[] data, ImageType type, String extension) in E:\ai\swarmui\SwarmUI\src\Utils\Image.cs:line 79 at SwarmUI.Utils.Image..ctor(String base64, ImageType type, String extension) in E:\ai\swarmui\SwarmUI\src\Utils\Image.cs:line 64 at SwarmUI.Text2Image.T2IParamInput.g__imageFor|55_1(String val) in E:\ai\swarmui\SwarmUI\src\Text2Image\T2IParamInput.cs:line 1316 at SwarmUI.Text2Image.T2IParamInput.Set(T2IParamType param, String val) in E:\ai\swarmui\SwarmUI\src\Text2Image\T2IParamInput.cs:line 1324 at SwarmUI.Text2Image.T2IParamTypes.ApplyParameter(String paramTypeName, String value, T2IParamInput data) in E:\ai\swarmui\SwarmUI\src\Text2Image\T2IParamTypes.cs:line 987 --- End of inner exception stack trace --- at SwarmUI.Text2Image.T2IParamTypes.ApplyParameter(String paramTypeName, String value, T2IParamInput data) in E:\ai\swarmui\SwarmUI\src\Text2Image\T2IParamTypes.cs:line 995 at SwarmUI.Text2Image.T2IPreset.ApplyTo(T2IParamInput user_input) in E:\ai\swarmui\SwarmUI\src\Text2Image\T2IPreset.cs:line 48 at SwarmUI.WebAPI.T2IAPI.RequestToParams(Session session, JObject rawInput) in E:\ai\swarmui\SwarmUI\src\WebAPI\T2IAPI.cs:line 227 at SwarmUI.WebAPI.T2IAPI.GenT2I_Internal(Session session, ValueTuple`4 input, Action`1 output, Boolean isWS) in E:\ai\swarmui\SwarmUI\src\WebAPI\T2IAPI.cs:line 254 --- End of inner exception stack trace ---

marcon kun

Go to advanced model addons and change vae to automatic. It seems it does not have vae downladed and will do it when as soon as you change this setting.

marcon kun

Have problem with installing RIFE. Basically it hangs for 2 minutes on fresh install than it closes with timeout error. At next run install button is not visible anymore and following errors show up (as RIFE folder is empty) : 23:20:52.773 [Warning] [ComfyUI-0/STDERR] Traceback (most recent call last): 23:20:52.775 [Warning] [ComfyUI-0/STDERR] File "D:\AI\SwarmUI_Model_Downloader_v39\SwarmUI\dlbackend\comfy\ComfyUI\nodes.py", line 2131, in load_custom_node 23:20:52.776 [Warning] [ComfyUI-0/STDERR] module_spec.loader.exec_module(module) 23:20:52.776 [Warning] [ComfyUI-0/STDERR] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ 23:20:52.777 [Warning] [ComfyUI-0/STDERR] File "", line 1022, in exec_module 23:20:52.778 [Warning] [ComfyUI-0/STDERR] File "", line 1159, in get_code 23:20:52.778 [Warning] [ComfyUI-0/STDERR] File "", line 1217, in get_data 23:20:52.778 [Warning] [ComfyUI-0/STDERR] FileNotFoundError: [Errno 2] No such file or directory: 'D:\\AI\\SwarmUI_Model_Downloader_v39\\SwarmUI\\src\\BuiltinExtensions\\ComfyUIBackend\\DLNodes\\ComfyUI-Frame-Interpolation\\__init__.py'

SECourses

it must be using a lot of shared VRAM. try Q4. i think SwarmUI not doing proper block swapping i asked this

Klaus The.

Thank you very much, can confirm with your new preset the installation now works like a charm!

Hockey

I thought that too and looked in the task manager>performance and it said 0.1 GB Shared VRAM. So I don't think that's the issue. Even if I used 100 steps at 1,024 x 1,024 VRAM never exceeded 16 GB and no shared VRAM was used according to task manager.

Ec Jep

In your opinion which is best and easiest to use for image to video generation: Framepack, Wan 2.1 I2V - V2V - T2V Advanced Gradio APP V70, or latest SwarmUI with WAN support (this post). So many options I don't know how to keep them separated in my work flow.

SECourses

i agree. unfortunately only way is testing all and prefer whichever you like :D

Dan

Thank you for this, was just thinking few days ago if it works to make video with Wan in SwarmUI with the Comfy back-end, to maximize the GPU and speed of generation, and then this post arrived 😊

Ec Jep

I can make consistent I2V using framepack latest, get only black I2V's from latest Wan2.1 install and get very grainy videos using latest swarmUI. I use the defaults so not having much success.

SECourses

i am gonna publish a video about teacache hopefully very soon and regular Wan 2.1 . i searched configs

Ec Jep

I'll try again tomorrow with Wan2.1 and SwarmUI - I use SwarmUI all the time with Flux T2I and absolutely love it.

Dan

Ok cool. One question, do you have any plans to do UI for some text2voice open source models or voice2voice? Might be interesting to do Ai audio. Cheers.

SECourses

yes i definitely want to do that and planning

SECourses

great. i am also about to publish new preset concluding the settings.