Home Creators Posts Import Register Favorites Logout

Content

Lo and behold, the build is nigh! :D 

As usual, thanks to the neverending support from you and meticulous testing by L and XL tiers!

Download notebook: https://www.patreon.com/posts/74675151

Here's the cumulative changelog: 

New stuff:

  • add normalize_latent modes: off, first_latent, user_defined
  • add normalize_latent user preset std and mean settings
  • add latent_norm_4d setting for per-channel latent normalization (was off in legacy colabs)
  • add colormatch_frame modes: off, init_frame, init_frame_offset, stylized_frame, stylized_frame_offset
  • add color match algorithm selection: LAB, PDF, mean (LAB was the default in legacy colabs)
  • add color match offset setting
  • add color match regrain flag
  • add color match strength
  • add resize to colormatching
  • add resize to colormatching :D
  • add cfg_scale schedule
  • add option to apply schedule templates to peak difference frames only
  • add flow multiplier (for glitches)
  • add flow remapping (for even more glitches)
  • add inverse mask
  • add background masks for both render and video export
  • add automatic mask extraction from video
  • add raw video input as a mask
  • add background input options (flat color, still image, init video)
  • unpack video frames folder into separate folders
  • add match first frame color toggle (was always on before)
  • add match first frame latent option
  • add karras noise support and ramp option
  • add negative prompts
  • add frame difference analysis
  • add difference plotting
  • add automated schedule templates
  • make persistent preview image
  • add schedules

Fixes: 

  • fix colormatching being forever on
  • fix masking in turbo mode
  • fix deleting videoframes not working in some cases
  • fix colormatching resizing wrong image
  • fix mask_video not extracting masked frames
  • move google drive init to the top
  • fix some bugs leading to unsaveable output frames


Normalize latent tweaks

normalize_latent
modes:
- off: disabled
- first_latent: legacy behavior, matches latent stats to the 1st stylized frame latent
- user_defined: matches to user defined stats in latent_fixed_std and latent_fixed_mean

latent_norm_4d: normalize on per-channel basis instead of 1 channel like it was before.

Color match frames tweaks

colormatch_frame
modes:
- off: disabled
- stylized_frame: match to fixed stylized frame (used in legacy)
- init_frame: match to fixed raw init video frame
- stylized_frame_offset: match to stylized frame with offset relative to current frame
- init_frame_offset: match to raw init video frame with offset relative to current frame

color_match_frame_str: color match opacity. 1 = full, 0 = off. (1 in legacy)
colormatch_offset: fixed frame number for stylized frame/init frame modes, offset back a few frames for offset modes (0 in legacy)
colormatch_method:
- LAB - match in LAB colorspace (used in legacy)
- PDF - match via probabilty density function
- mean - match by mean
colormatch_regrain: restore texture from reference frame (off in legacy)


Cfg_scale schedule

Cfg_scale can now be scheduled manually and automatically via template

Apply schedule templates to peak difference frames only

Templates can now respect manual schedules. That means that the template will override manual schedules only at keyframes (the ones which difference is above threshold)

Flow multiplier

Multiplies flow map values, moving pixels further than expected. Can provide interesting glitchy results.

Flow remapping

Lets you schedule flow maps. Usually, frame #1 takes flow map #1. You can now change that mapping to anything you want. For example, you can repeat certain flow maps across multiple frames to get continuous glitchy motion.

You will need to get a little familiar with python ranges and list manipulations for that.

The basics: [0.5] is a list with 0.5 as its 0 element. [0.5, 0.2] has 2 elements, 0.5 at position 0, and 0.2 at 1.

You can add lists. [0.5] + [0.2] = [0.5,0.2]

You can multiply lists. [0.5]*3 + [0.2] = [0.5, 0.5, 0.5] + [0.2] = [0.5, 0.5, 0.5, 0.2]

You can use range iterator to fill long lists: [*range(0,10)] creates a list from 0 to 10: [0,1,2,3,4,5,6,7,8,9,10]

Inverse mask

Inverts background mask, allowing to stylize only the background instead of the foreground character.

Video Masking

mask_source: init_video, mask_video :
Generate (extract) background mask from your init video or any other video, or use a raw video as a mask

extract_background_mask: Check to rotoscope the video and create a mask from it. If unchecked, the raw monochrome video will be used as a mask.

mask_video_path: Specify path to a mask source video for mask_video mode.

Video masking (render-time)

use_background_mask:
Check to enable background masking during render. Use with care, because model will react to new injected background, possibly producing unexpected results, better use masking when creating the output video for more control and faster testing.

apply_mask_after_warp:
Apply mask right before feeding init image to the model. Unchecking will only mask current raw init frame before warping and blending with previous frame.

background:
Choose background source to paste masked stylized image onto: image, color, init video.

background_source:
Specify the init image path or color depending on your background source choice.

Frame correction

normalize_first_latent: match the range of latent vector towards the 1st frame range. Doesn't restrict colors, but may limit contrast.

colormatch_first_frame:
color match frame towards the 1st stylized frame. Helps prevent images going deep purple. As a drawback, locks colors to the 1st frame. Turn this one to reproduce previous notebooks.

use_karras_noise:
Should work better with current sample, needs more testing.

end_karras_ramp_early: controls karras noise schedule


Content-aware scheduing:
analyze_video: turn on frame comparison

diff_function:
rmse - root mean squared error. Basic Function, won't work well in scenes where the colors are close, but the content is different.
lpips - compares features, slower but better.
rmse+lpips: a combo, slower, not neccessarily better.

After running this cell you'll get a plot of frame differences, and a suggested threshold (blue line) to use when creating schedules. You can try your own threshold in the next cell- Plot threshold vs frame difference, make sure it covers all the peaks and doesn't cross other values.

Create schedules from frame difference

Creates schedules from frame difference, based on the template you input below. If you leave the input empty, the template from stable-settings cells will be used.

Template format:

[normal value, high difference value, difference threshold, falloff from high to normal (number of frames)]

For example, setting flow_blend template to [0.999, 0.3, 0.5, 5] will use 0.999 flow_blend value everywhere unless a scene has changed (frame difference >threshold, which is 0.5 here) and then set flow_blend for this frame to 0.3 and gradually fade to 0.999 in 5 frames


Schedules 

I've began with replacing skip steps and frames_skip_steps with more widespread strength setting and its respective schedule, so that newcomers will be able grasp the concept more easily.

The schedules are available for flow_blend and skip_steps / frames_skip_steps for now. Turbo skip steps work the old way, need to think how to universally implement scheduling for those, too.

style_strength_schedule: controls the skip steps setting. 1 means 0% (no) skip steps, 0 means 100% skip steps, 0.3 means 70% skip steps,  etc.

schedule is a python list, so you can do all the usual stuff like this: [0.5]*3+[0.2] to use 0.5 strength / 50% skip steps for the first 3 frames and the use 0.2 strength / 80% skip steps for the rest of the frames.

A simple list like [0.5, 0.2] mimics the legacy behavior with 50% skip_steps and 80% frames_skip_steps.

flow_blend_schedule works in a similar way, but mind that it start blending frames at frame 1, so the flow_blend_schedule = [0.5,0.1] will skip 0.5 and use 0.1 for the blend between the 1st and the 2nd frames.

I will be adding more schedules to get rid of all those "first frame / all other frames" settings, as those may be confusing.

upd: move flow_blend_schedule defaults back to 0.999

upd: add steps, init_scale, init_latent_scale schedules. Scheduling steps is useful with strength scheduling, when you end up with low strength and thus very few diffusion steps. For example, 0.2 strength at 50 total steps = 10 diffusion steps, which may be too little with animated prompts. Setting steps higher for frames with low strength lets us have more steps without losing coherence.


Download notebook: https://www.patreon.com/posts/74675151