SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts

FLUX De-Distilled and Anti-Bleeding Fine-Tuning / DreamBooth & LoRA Training Experiments - Also Testing CFG Impact for Stylized Images (Patreon)

Published:

2024-10-31 12:28:00

Edited:

2024-10-31 14:48:00

Imported:

2026-06

Content

Patreon exclusive posts index to find our scripts easily, Patreon scripts updates history to see which updates arrived to which scripts and amazing Patreon special generative scripts list that you can use in any of your task.

Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord

Please also Star, Watch and Fork our Stable Diffusion & Generative AI GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)

=======

To understand this post, you need have fully watched this tutorial : https://youtu.be/FvpWy1x5etM

You need to have fully read this post : https://www.patreon.com/posts/112099700

When training with de-distilled models like (consolidated_s6700.safetensors or VerusVision_1.0b_Transformer_fp16.safetensors), what you need to change is from our FLUX Fine-Tuning or LoRA training configs is only single parameter which is Guidance Scale

You need to change it to 3.5 from 1

If you are training community Fine-Tuned models like PixelWave which is not De-Distilled, use and train it same way as training base FLUX Dev model

I have tested 1, 2.5 and 3.5 and 3.5 yields best performance as can be seen in attached grids below on De-Distilled model

Moreover, when doing inference, we set regular CFG to 3.5 like in Stable Diffusion models instead of setting as 1 for regular FLUX DEV models

As tested in below grids, CFG 3.5 yields best quality compared to 2.5, 3.0 for de-distilled models

Using Higher CFG on FLUX DEV for Stylized Images

Another finding of myself is that training on regular FLUX DEV model as usual but when doing inference, not using CFG Scale 1 but using higher CFG Scale when generating stylized images like anime or painting

As can be seen in grids, this is either hit or miss.

Again CFG 3.5 looks like a good value to test

Complete Test Results and Findings

I have used 28 images selected from sub-set of my 256 images to train myself

I have used Dwayne Johnson 28 images dataset fully shared here : https://www.patreon.com/posts/114972274

I have tested with the following models

FLUX DEV - base model - CFG 1

https://huggingface.co/black-forest-labs/FLUX.1-dev

Verus Vision 1.0b - de-distilled model - CFG 3.5

https://huggingface.co/SG161222/Verus_Vision_1.0b

PixelWave v03 - Fine-Tuned model - CFG 1

https://civitai.com/models/141592/pixelwave?modelVersionId=992642

FLUX DEV-De-Distilled - CFG 3.5

https://huggingface.co/nyanko7/flux-dev-de-distill

You can download all above models from below repository

https://huggingface.co/OwlMaster/SD3New/tree/main

Conclusions

PixelWave is overfit and bad quality model for realism training

Verus Vision not working better than DEV-De-Distilled - don't see reason to use it

FLUX DEV-De-Distilled still not fixes the bleeding / mixing problem when training multiple-concept sadly

Moreover, still your training overwrites your trained subject's class, such as if you are training a man, all man becomes your trained man, you captioned or not doesn't matter since FLUX has internal auto captioning alike mechanism - image embeddings used

Or lets say I trained myself and Dwayne Johnson in the same training

My images were captioned as only ohwx

Dwayne Johnson images were captioned only as bbuk

The mixing / bleeding issue still happens at DEV-De-Distilled model

Looking at the full grids will give you exact idea

Yet DEV-De-Distilled model has lesser degree of bleed / mix problem compared to FLUX DEV model

DEV-De-Distilled is a good model as a standalone single concept training and you can see grids to decide yourself

When you train without class token, e.g. man, when you generate images sometimes it generates feminine image even if you are training a man, thus class token usage really important

For example on some images when generating with only ohwx or bbuk, it was generating woman or woman like images - the trainings were also made without class token to reduce bleeding / mixing problem

However, even though I didn't use class token, still it did bleed due to internal captioning system of FLUX and FLUX being originally distilled model

Sadly de-distilled models not reached full model level yet

Hopefully I will fully research SD 3.5 to see if it works better for multi-concept and how to train it like FLUX