Stable WarpFusion v0.5.28 Nightly - captions - notebook (Patreon)
Downloads
Missing 2 files, as this post has not yet been imported.
Content
Hello, my fellow warp explorers!
Content-aware conditioning is now expanded with automatic caption generation, and you can now use separate video sources for depth maps and inpainting masks, colormatching.
Here's the changelog for v0.5.28:
New Stuff:
- add conditioning source video for depth, inpainting models
- add conditioning source to gui
- add automatic caption generation
- add caption syntax to prompts
- add separate color video / image as colormatch source
- add color video to gui options
- save settings to settings folder
Fixes:
- fix blip config path
- shift caption frame number
- convert loaded setting keys to int ("0" -> 0)
- fix batchnum not inferred correctly due to moved settings files
- fix init frame / stylized frame colormatch with offset 0 error
Upd:
- fix ANSI encoding error
- fix videoFramesCaptions error when captions were off
- fix caption keyframes "nonetype is not iterable" error
- change default colormatch mode to stylized frame to avoid filenotfound error
- fix colormatch first frame error
- fix raft_model not found error when generate flow cell is run after the cell with warp_towards_init
- fix lbip_moder nor found error
Image conditioning source video
Video Input Settings -> Image Conditioning Video Source
This is a special video source for image-conditioned models, such as inpainting and depth.
You can now specify a separate video source for those models. For example, you can use alpha mask as depth source to create a cardboard-like animation
To acctivate conditioning source video, select
GUI -> depth_source / inpainting_mask_source -> color_video option, depending on the model used. This setting will have no efffect with non-depth and non-inpainting models.
Caption generation
Generate frame captions -> Generate captions for keyframes
In order to further extend the content-aware automatic settings feature, I've added caption generation. You can now caption frames based on various keyframe settings.
Captions are saved in videoFramesCaptions folder inside your output folder, so you can verify and correct captions manually.
make_captions:
Enable caption generation during run-all-cells
keyframe_source:
Content-aware scheduling keyframes
- captions frames at keyframes, using frame difference detected in the Content-aware scheduling cell
User-defined keyframe list
- you can specify a list of frame numbers manually
Every n-th frame
- generate caption for every n-th frame, starting with the 1st frame.
user_defined_keyframes:
Used only with keyframe_source = User-defined keyframe list.
Specify a list of keyframes to generate captions for.
diff_thresh:
Used only with keyframe_source = Content-aware scheduling keyframes.
Defines threshold difference above which will trigger caption generation.
nth_frame:
Used only with keyframe_source = Every n-th frame.
Generates caption for every n-th frame, starting with the 1st frame.
offset_mode:
Fixed
- Fixed offset. If you have keyframes at frames 0 and 10, fixed_offset 1, caption wil be generated for frames 1 and 11.
Between Keyframes
- Captions are generated between keyframes. If you have keyframes at frames 0 and 10, caption wil be generated for frame 5.
None
- No offset
fixed_offset:
Used only with offset_mode = fixed. Shifts captioned frame number with an offset.
Using Captions in Prompts
You can call caption in you prompt by using the {caption} keyword. It will read currect frame's caption, or the latest caption if current frame doesn't have one.
You can mix prompt schedule with dynamic captions, for example have multiple caption with one prompt or vice versa.
External colormatch / norm source
Video Input Settings -> Colormatch -> Colormatching Video Source
Added external colormatching / norm source video / frame option.
You can specify a video or a single image as color palette source.
For this setting to work well, also enable color_match in consistency tab.
To use colormatching video enable either or both of those:
GUI -> colormatch_frame -> color_video & color_video_offset
GUI -> normalize_latent -> color_video & color_video_offset
Enabling colormatch_after can also be useful.
Save settings to settings folder
Settings' files are now saved to settings subfolder. Please run this notebook with a new batchname so that run numbers will be detected correctly.
A reminder:
Changes in GUI will not be saved into the notebook, but if you run it with new settings, they will be saved to a settings.txt file as usual.
You can load settings in misc tab.
You do not need to rerun the GUI cell after changing its settings.
Local install guide:
https://discord.com/channels/973802253204996116/1067887206125015153/1067888756910215278
https://github.com/Sxela/WarpFusion/blob/main/README.md
Youtube playlist with settings:
https://www.youtube.com/watch?v=wvvcWm4Snmc&list=PL2cEnissQhlCUgjnGrdvYMwUaDkGemLGq
For tech support and other questions please join our discord server:
https://discord.gg/YrpJRgVcax
Discord is the preferred method, because it is nearly impossible to provide any decent help or tech support via Patreon due its limited text formatting and inability to add screenshots or videos to comments or DMs.
Error reports in comments will be deleted and reposted in discord.