sxela

Stable WarpFusion v0.5.195 Nightly - frame matching tweaks (Patreon)

Published:

2022-11-04 15:34:46

Imported:

2026-06

Tags:

colab nightly notebook text2video warp warpfusion

haven't archived this post yet. have a subscription? use the importer!

Downloads

stable_warpfusion_v0_5_19_10_matchingstuff.ipynb

Missing 1 full-res photo, 1 file, as this post has not yet been imported.

Content

Another update to v0.5.19, introducing some frame matching tweaks.

Frame matching manages stylized frames' post-render adjustments, that is aimed to reduce color bleeding andringing/feedback loop artifacts.

With colormatch_frames and normalize_latent both off the output will go deep purple in just a few frames, probably dur to a feedback loops being introduced due to images being fed back and forth in a loop into stable diffusion autoencoder, and then clipped to RGB range afterwards. Before investigating this furter, I've added a few tweaks that can already help with the issue.

Changelog:

- add normalize_latent modes: off, first_latent, user_defined
- add normalize_latent user preset std and mean settings
- add latent_norm_4d setting for per-channel latent normalization (was off in legacy colabs)
- add colormatch_frame modes: off, init_frame, init_frame_offset, stylized_frame, stylized_frame_offset
- add color match algorithm selection: LAB, PDF, mean (LAB was the default in legacy colabs)
- add color match offset setting
- add color match regrain flag
- add color match strength

Normalize latent tweaks

normalize_latent modes:
- off: disabled
- first_latent: legacy behavior, matches latent stats to the 1st stylized frame latent
- user_defined: matches to user defined stats in latent_fixed_std and latent_fixed_mean

latent_norm_4d: normalize on per-channel basis instead of 1 channel like it was before.

Color match frames tweaks

colormatch_frame modes:
- off: disabled
- stylized_frame: match to fixed stylized frame (used in legacy)
- init_frame: match to fixed raw init video frame
- stylized_frame_offset: match to stylized frame with offset relative to current frame
- init_frame_offset: match to raw init video frame with offset relative to current frame

color_match_frame_str: color match opacity. 1 = full, 0 = off. (1 in legacy)
colormatch_offset: fixed frame number for stylized frame/init frame modes, offset back a few frames for offset modes (0 in legacy)
colormatch_method:
- LAB - match in LAB colorspace (used in legacy)
- PDF - match via probabilty density function
- mean - match by mean
colormatch_regrain: restore texture from reference frame (off in legacy)

upd: add resize to colormatching
upd: fix colormatching being forever on
upd: add resize to colormatching :D
upd: fix colormatching resizing wrong image
upd: fix mask_video not extracting masked frames

Files

Previews only

Stable WarpFusion v0.5.195 Nightly - frame matching stuff

Settings: { "text_prompts": { "0": [ "a beautiful breathtaking highly-detailed intricate portrait painting of a young bodybuilder elon musk, in the style of alphonse mucha and ilya kuvshinov and peter mohrbacker, ross tran rossdraws, watercolor, featured on artstation, 70mm, rendered in octane" ], "550": [ "a highly detailed matte painting of a robot, No Man's Sky Screenshot, tall grass field, broken machinery, Simon Stalenhag , featured on Artstation." ], "1000": [ "a beautiful breathtaking highly-detailed intricate portrait painting of a blissful ignorant enlightened young sxelaqwertyop dancing against a backdrop of stars in the style of alphonse mucha and ilya kuvshinov and peter mohrbacker, ross tran rossdraws, watercolor, featured on artstation, 70mm, rendered in octane" ], "1500": [ "a highly detailed matte painting of a robot sxelaqwertyop, No Man's Sky Screenshot, tall grass field, broken machinery, Simon Stalenhag , featured on Artstation." ] }, "user_comment": "testing cc layers", "image_prompts": {}, "range_scale": 0, "sat_scale": 0, "max_frames": 50, "interp_spline": "Linear", "init_image": "/content/photo_2022-10-13_12-25-34.jpg", "clamp_grad": true, "clamp_max": 0.7, "seed": 4275770367, "fuzzy_prompt": false, "rand_mag": 0.05, "eta": 0.55, "width": 512, "height": 704, "diffusion_model": "stable_diffusion", "diffusion_steps": 1000, "video_init_path": "/content/out.mp4", "extract_nth_frame": 1, "flow_video_init_path": null, "flow_extract_nth_frame": 2, "video_init_seed_continuity": false, "turbo_mode": false, "turbo_steps": "3", "turbo_preroll": 1, "flow_warp": true, "check_consistency": true, "turbo_frame_skips_steps": null, "forward_weights_clip": 0, "forward_weights_clip_turbo_step": 0, "padding_ratio": 0.2, "padding_mode": "reflect", "consistency_blur": 2, "inpaint_blend": 0, "match_color_strength": 0, "high_brightness_threshold": 180, "high_brightness_adjust_ratio": 0.97, "low_brightness_threshold": 40, "low_brightness_adjust_ratio": 1.03, "stop_early": 0, "high_brightness_adjust_fix_amount": 2, "low_brightness_adjust_fix_amount": 2, "max_brightness_threshold": 254, "min_brightness_threshold": 1, "enable_adjust_brightness": false, "dynamic_thresh": 50, "warp_interp": 1, "fixed_code": false, "blend_code": 1, "normalize_code": false, "mask_result": false, "reverse_cc_order": true, "flow_lq": true, "use_predicted_noise": false, "clip_guidance_scale": 0, "clip_type": "ViT-H-14", "clip_pretrain": "laion2b_s32b_b79k", "missed_consistency_weight": 1, "overshoot_consistency_weight": 1, "edges_consistency_weight": 1, "style_strength_schedule": [ 0.5, 0.2 ], "flow_blend_schedule": [ 0.999 ], "steps_schedule": [ 50, 70 ], "init_scale_schedule": [ 0, 0 ], "latent_scale_schedule": [ 0, 0 ], "latent_scale_template": "", "init_scale_template": "", "steps_template": "", "style_strength_template": [ 0.2, 0.5, 0.55, 3 ], "flow_blend_template": [ 0.999, 0.0, 0.55, 3 ], "make_schedules": false, "normalize_latent": "off", "colormatch_frame": "init_frame_offset", "use_karras_noise": false, "end_karras_ramp_early": false, "use_background_mask": true, "apply_mask_after_warp": true, "background": "init_video", "background_source": "red", "mask_source": "init_video", "extract_background_mask": true, "mask_video_path": "", "negative_prompts": { "0": [ "text, logo, cropped, two heads, four arms, lazy eye, blurry, unfocused" ] }, "invert_mask": false, "warp_strength": 1, "flow_override_map": [], "cfg_scale_schedule": [ 15 ], "respect_sched": true, "color_match_frame_str": 1, "colormatch_offset": 0, "latent_fixed_mean": 0.0, "latent_fixed_std": 0.9, "colormatch_method": "mean", "colormatch_regrain": true }