DreamBooth Training on SDXL Anime Model & Comparing Ground Truth Regularization Images Effect (Patreon)

Patreon exclusive posts index

Join discord and tell me your discord username to get a special rank : SECourses Discord

It was requested of me to test ground truth Regularization / Classification images effect during Stable Diffusion XL (SDXL) DreamBooth training.

So for this experimentation I have used an anime SDXL model name as : Starlight XL 星光 Animated V3.

I used the configuration I shared here currently. It has been discovered after 120 different full trainings experimentation : https://www.patreon.com/posts/very-best-for-of-89213064

Hopefully I will make a full big tutorial of DreamBooth training and show all the settings.

For training Kohya SS GUI is used. If you don’t know how to install and use Kohya SS GUI here the tutorial you need > https://youtu.be/sBFGitIvD2A

And here is the tutorial that shows how to use shared configuration both on Windows and RunPod > https://www.youtube.com/watch?v=EEV8RPohsbw

So I used the below images as training images dataset.

This training dataset is at best medium quality because it has repeating backgrounds, repeating clothing and missing fully body pose.

Then for training configuration I did the following.

When training with regularization images, I did 1 epoch and 150 repeating. So more regularization images are used. And saved checkpoints at every 30 epochs. To make it that way calculate the number of steps you need to save. It is easy. 15 * 30 * 2 + 1

When training without regularization images, i set the repeating 1, trained up to 150 epochs and saved checkpoints at every 30 epoch. That is also why model names has different format. One has epoch number 1 has steps number.

I have used ultra high quality manually picked Unsplash real images having man dataset as regularization images dataset. This means they were ground truth regularization images like how Stable Diffusion was initially trained.

You can download dataset from > https://www.patreon.com/posts/massive-4k-woman-87700469

5200 images for both man and woman. Perfect quality cropped and resized to many common resolutions.

All training images and reg images were 1024x1024 pixels.

I used the 275 Fooocus amazing styles file to generate images with models so that I could pick later on.

You can find styles file (styles . csv) and styles file generator script here : https://www.patreon.com/posts/all-amazing-of-95143823

I saved both After Detailer (ADetailer) before and after images. So some images are ADetailer used and some not. This is in settings panel to set both. For after detailer I used 0.7 inpainting denoise and 80 steps manually set. Improves quality.

And here the results.

First I will compare anime

The png info as below. I used batch PNG info extractor to get them easier : https://www.patreon.com/posts/batch-image-for-95176238

parameters: picture of (ohwx man:1.1) with shirt in anime style, anime movie, anime cartoon, anime battle, muscular, anime screenshot
Negative prompt: shirtless,topless,nsfw
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 9, Seed: 1382869474, Size: 1024x1024, Model hash: f31347de70, Model: me_15_StarlightXLAnimatedV3_noREG-000030, ADetailer model: face_yolov8n.pt, ADetailer confidence: 0.3, ADetailer mask only top k largest: 1, ADetailer dilate erode: 4, ADetailer mask blur: 4, ADetailer denoising strength: 0.5, ADetailer inpaint only masked: True, ADetailer inpaint padding: 32, ADetailer use separate steps: True, ADetailer steps: 80, ADetailer version: 23.11.1, Script: X/Y/Z plot, X Type: Checkpoint name, X Values: "me_15_StarlightXLAnimatedV3_noREG-000030.safetensors [f31347de70],me15img\\model\\me_15_StarlightXLAnimatedV3-step00000901.safetensors [4610bf5599]", Version: v1.7.0-133-gde03882d

As you can see above, the regularization images affect is making it more like us. However, it reduced the stylization capability. It adds realism and more likeliness.

Now here second prompt compared. This one is realistic image instead of anime.

parameters: luxury product style portrait of ohwx man . elegant, sophisticated, high-end, luxurious, professional, highly detailed
Negative prompt: cheap, noisy, blurry, unattractive, amateurish
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 2811753849, Size: 1024x1024, Model hash: f31347de70, Model: me_15_StarlightXLAnimatedV3_noREG-000030, ADetailer model: face_yolov8n.pt, ADetailer confidence: 0.3, ADetailer mask only top k largest: 1, ADetailer dilate erode: 4, ADetailer mask blur: 4, ADetailer denoising strength: 0.7, ADetailer inpaint only masked: True, ADetailer inpaint padding: 32, ADetailer use separate steps: True, ADetailer steps: 80, ADetailer version: 23.11.1, Script: X/Y/Z plot, X Type: Checkpoint name, X Values: "me_15_StarlightXLAnimatedV3_noREG-000030.safetensors [f31347de70],me15img\\model\\me_15_StarlightXLAnimatedV3-step00000901.safetensors [4610bf5599]", Version: v1.7.0-133-gde03882d

As can be seen above again, the reg images improved likeliness and realism quality.

And here some other good images I generated

parameters: retro game art portrait of ohwx man . 16-bit, vibrant colors, pixelated, nostalgic, charming, fun
Negative prompt: realistic, photorealistic, 35mm film, deformed, glitch, low contrast, noisy
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 2542262511, Size: 1024x1024, Model hash: f8f06f0d73, Model: me_15_StarlightXLAnimatedV3-step00002703, ADetailer model: face_yolov8n.pt, ADetailer confidence: 0.3, ADetailer mask only top k largest: 1, ADetailer dilate erode: 4, ADetailer mask blur: 4, ADetailer denoising strength: 0.7, ADetailer inpaint only masked: True, ADetailer inpaint padding: 32, ADetailer use separate steps: True, ADetailer steps: 80, ADetailer version: 23.11.1, Version: v1.7.0-133-gde03882d

parameters: breathtaking illustration from adult comic book presenting portrait of ohwx man. fabulous artwork. best quality, high resolution
Negative prompt: deformed, ugly, low quality, low resolution
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 427502473, Size: 1024x1024, Model hash: f8f06f0d73, Model: me_15_StarlightXLAnimatedV3-step00002703, Version: v1.7.0-133-gde03882d

parameters: closeshot picture of (ohwx man:1.1) wearing shirt, 3D render, 3D art, 3d character, muscular game, cgi, artstation
Negative prompt: ugly, deformed, noisy, blurry, low contrast, grunge, sloppy, unkempt, photograph, photo, realistic, nsfw, shirtless, topless
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 1171263453, Size: 1024x1024, Model hash: f8f06f0d73, Model: me_15_StarlightXLAnimatedV3-step00002703, Version: v1.7.0-133-gde03882d

parameters: picture of (ohwx man:1.1) in Photorealistic chibi art style,   facing the viewers , artistic, artist, simple volumetric light background, swirling clouds, ColorART, many cute ghosts with tongues out, chibi,
Negative prompt: gaussian noise worst quality lowres oversaturated undersaturated overexposred underexposed grayscale bw bad photo bad photography bad art blur blurry grainy morbid ugly asymmetrical mutated malformed mutilated poorly lit bad shadow draft cropped out of frame cut off censored jpeg artifacts out of focus glitch duplicate pixelated soft focus color fringing overprocessed oversharpened
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 3576768970, Size: 1024x1024, Model hash: f8f06f0d73, Model: me_15_StarlightXLAnimatedV3-step00002703, Version: v1.7.0-133-gde03882d

parameters: portrait of (ohwx man:1.1), painting by sargent and rhads and leyendecker and greg hildebrandt evening sky
Negative prompt: gaussian noise worst quality lowres oversaturated undersaturated overexposed underexposed grayscale bw bad photo bad photography bad art blur blurry grainy morbid ugly asymmetrical mutated malformed mutilated poorly lit bad shadow draft cropped out of frame cut off censored jpeg artifacts out of focus glitch duplicate pixelated soft focus color fringing overprocessed oversharpened
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 1252309971, Size: 1024x1024, Model hash: f8f06f0d73, Model: me_15_StarlightXLAnimatedV3-step00002703, Version: v1.7.0-133-gde03882d

parameters: Watercolor style painting, portrait of ohwx man , visible paper texture, colorwash, watercolor
Negative prompt: ugly, deformed, noisy, blurry, low contrast, photo, realistic
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 3204333295, Size: 1024x1024, Model hash: 29bbb5ce43, Model: me_15_StarlightXLAnimatedV3-step00001802, ADetailer model: face_yolov8n.pt, ADetailer confidence: 0.3, ADetailer mask only top k largest: 1, ADetailer dilate erode: 4, ADetailer mask blur: 4, ADetailer denoising strength: 0.7, ADetailer inpaint only masked: True, ADetailer inpaint padding: 32, ADetailer use separate steps: True, ADetailer steps: 80, ADetailer version: 23.11.1, Version: v1.7.0-133-gde03882d

parameters: Flat 2D Art, portrait of ohwx man , simple flat color, 2-dimensional, Flat 2D Art Style
Negative prompt: ugly, deformed, noisy, blurry, low contrast, 3D, photo, realistic
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 998589366, Size: 1024x1024, Model hash: f8f06f0d73, Model: me_15_StarlightXLAnimatedV3-step00002703, Version: v1.7.0-133-gde03882d

parameters: pixel-art portrait of ohwx man . low-res, blocky, pixel art style, 8-bit graphics
Negative prompt: sloppy, messy, blurry, noisy, highly detailed, ultra textured, photo, realistic
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 3068779067, Size: 1024x1024, Model hash: f8f06f0d73, Model: me_15_StarlightXLAnimatedV3-step00002703, Version: v1.7.0-133-gde03882d

parameters: comic portrait of ohwx man . graphic illustration, comic art, graphic novel art, vibrant, highly detailed
Negative prompt: photograph, deformed, glitch, noisy, realistic, stock photo
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 2542262511, Size: 1024x1024, Model hash: f8f06f0d73, Model: me_15_StarlightXLAnimatedV3-step00002703, ADetailer model: face_yolov8n.pt, ADetailer confidence: 0.3, ADetailer mask only top k largest: 1, ADetailer dilate erode: 4, ADetailer mask blur: 4, ADetailer denoising strength: 0.7, ADetailer inpaint only masked: True, ADetailer inpaint padding: 32, ADetailer use separate steps: True, ADetailer steps: 80, ADetailer version: 23.11.1, Version: v1.7.0-133-gde03882d

parameters: breathtaking illustration from adult comic book presenting portrait of ohwx man. fabulous artwork. best quality, high resolution
Negative prompt: deformed, ugly, low quality, low resolution
Steps: 20, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 427502468, Size: 1024x1024, Model hash: f8f06f0d73, Model: me_15_StarlightXLAnimatedV3-step00002703, Version: v1.7.0-133-gde03882d