sxela

Stable WarpFusion v0.5.18-nightly - frame comparison (Patreon)

Published:

2022-10-12 15:39:01

Imported:

2026-06

Tags:

notebook text2video warp warpfusion

haven't archived this post yet. have a subscription? use the importer!

Downloads

stable_warpfusion_v0_5_18_frame_difference.ipynb

Missing 1 full-res photo, 1 file, as this post has not yet been imported.

Content

An expected successor to the schedules' update, this build allows us to compare init video frames, detect high difference areas, and automatically create schedules to stylize those frames better.

Example video: cc means consistency channel mixing settings. noshed means schedules from stable-settings are being used, autosched - schedules were created from frame difference.

Changes:

- add frame difference analysis
- add difference plotting
- add automated schedule templates
- make persistent preview image
- fix some bugs leading to unsaveable output frames

New cells:
Content-aware scheduing
Plot threshold vs frame difference
Create schedules from frame difference

Content-aware scheduing:
analyze_video: turn on frame comparison

diff_function:
rmse - root mean squared error. Basic Function, won't work well in scenes where the colors are close, but the content is different.
lpips - compares features, slower but better.
rmse+lpips: a combo, slower, not neccessarily better.

After running this cell you'll get a plot of frame differences, and a suggested threshold (blue line) to use when creating schedules. You can try your own threshold in the next cell- Plot threshold vs frame difference, make sure it covers all the peaks and doesn't cross other values.

Create schedules from frame difference

Creates schedules from frame difference, based on the template you input below. If you leave the input empty, the template from stable-settings cells will be used.

Template format:

[normal value, high difference value, difference threshold, falloff from high to normal (number of frames)]

For example, setting flow_blend template to [0.999, 0.3, 0.5, 5] will use 0.999 flow_blend value everywhere unless a scene has changed (frame difference >threshold, which is 0.5 here) and then set flow_blend for this frame to 0.3 and gradually fade to 0.999 in 5 frames

Don't hesitate to share your results and questions on discord. Good luck with your experiments!

Files

Previews only

Stable WarpFusion v0.5.18 Nightly - frame difference

Stable WarpFusion v0.5.18 Nightly - frame difference-based settings' scheduling Settings: { "text_prompts": { "0": [ "a highly detailed matte painting of a derelict war machine, No Man's Sky Screenshot, a tall grass field, broken machinery, Simon Stalenhag , featured on Artstation." ], "60": [ "a highly detailed matte painting of a robot, No Man's Sky Screenshot, a tall grass field, broken machinery, Simon Stalenhag , featured on Artstation." ] }, "user_comment": "testing cc layers", "image_prompts": {}, "range_scale": 0, "sat_scale": 0, "max_frames": 211, "interp_spline": "Linear", "init_image": "", "clamp_grad": true, "clamp_max": 0.7, "seed": 4275770367, "fuzzy_prompt": false, "rand_mag": 0.05, "eta": 0.55, "width": 768, "height": 512, "diffusion_model": "stable_diffusion", "diffusion_steps": 1000, "video_init_path": "/content/drive/MyDrive/deep_learning/input/Jurassic Park Trailer.mp4", "extract_nth_frame": 2, "flow_video_init_path": null, "flow_extract_nth_frame": 2, "video_init_seed_continuity": false, "turbo_mode": false, "turbo_steps": "3", "turbo_preroll": 1, "flow_warp": true, "check_consistency": true, "turbo_frame_skips_steps": null, "forward_weights_clip": 0, "forward_weights_clip_turbo_step": 0, "padding_ratio": 0.2, "padding_mode": "reflect", "consistency_blur": 2, "inpaint_blend": 0, "match_color_strength": 0, "high_brightness_threshold": 180, "high_brightness_adjust_ratio": 0.97, "low_brightness_threshold": 40, "low_brightness_adjust_ratio": 1.03, "stop_early": 0, "high_brightness_adjust_fix_amount": 2, "low_brightness_adjust_fix_amount": 2, "max_brightness_threshold": 254, "min_brightness_threshold": 1, "enable_adjust_brightness": false, "cfg_scale": 15, "dynamic_thresh": 2, "warp_interp": 1, "fixed_code": false, "blend_code": 1, "normalize_code": false, "mask_result": false, "reverse_cc_order": true, "flow_lq": true, "use_predicted_noise": false, "clip_guidance_scale": 0, "clip_type": "ViT-H-14", "clip_pretrain": "laion2b_s32b_b79k", "missed_consistency_weight": 1, "overshoot_consistency_weight": 1, "edges_consistency_weight": 1, "style_strength_schedule": [(omitted)], "flow_blend_schedule": [(omitted)], "steps_schedule": [ 50, 70 ], "init_scale_schedule": [ 0, 0 ], "latent_scale_schedule": [ 0, 0 ], "latent_scale_template": "", "init_scale_template": "", "steps_template": "", "style_strength_template": [ 0.2, 0.5, 0.55, 3 ], "flow_blend_template": [ 0.999, 0.0, 0.55, 3 ], "make_schedules": true }