Kohya FLUX LoRA and Fine Tuning Training Full Tutorial For Local Windows and Cloud RunPod and Massed Compute for Research & Development (Patreon)

THIS POST IS DEPRECATED NEW POST FOR DREAMBOOTH TRAINING / FINE TUNING : https://www.patreon.com/posts/112099700

THIS POST IS DEPRECATED NEW POST FOR LORA TRAINING : https://www.patreon.com/posts/110879657

FLUX DREAMBOOTH / FINE TUNING TRAINING FULL TUTORIAL : https://www.youtube.com/watch?v=FvpWy1x5etM

FLUX LORA TRAINING FULL TUTORIAL : https://www.youtube.com/watch?v=nySGu12Y05k

THIS POST IS DEPRECATED NEW POST FOR DREAMBOOTH TRAINING / FINE TUNING : https://www.patreon.com/posts/112099700

THIS POST IS DEPRECATED NEW POST FOR LORA TRAINING : https://www.patreon.com/posts/110879657

FLUX DREAMBOOTH / FINE TUNING TRAINING FULL TUTORIAL : https://www.youtube.com/watch?v=FvpWy1x5etM

FLUX LORA TRAINING FULL TUTORIAL : https://www.youtube.com/watch?v=nySGu12Y05k

==============================================

Full Zero to Hero Masterpiece FLUX LoRA and Full Fine Tuning Tutorial and Workflows and Scripts and Presets for Local Windows and Cloud

I will update this page as I progress. I am just starting so I will add the info. Nothing ready yet

FOLLOW THIS THREAD NOW FOR TUTORIAL : https://www.patreon.com/posts/110879657

Who has fbgemm.dll error on Windows 11 the solutions posted below topic

https://github.com/Acly/krita-ai-diffusion/issues/965

Quick solution is as below (verified DLL)

Download this dll and copy it to your /windows/system32 folder. https://www.dllme.com/dll/files/libomp140_x86_64/00637fe34a6043031c9ae4c6cf0a891d/download

Latest Zip file here : Kohya_GUI_Flux_Installer_27.zip

30 August 2024

Clip Large training research still in work

I have trained FLUX Schnell Turbo model and surprisingly it worked however results are not great

You can see full grid here : Flux_Schnell_Turbo_Test.jpg

I also trained with 4x GPU, used our config and enabled T5 masked attention too and the results are amazing

Almost linear speed gain

You can see full grid here : 4x_GPU_Results.jpg

28 August 2024

I have started training Clip L text encoder

Early experiments are promising I am going to do full research and update configs hopefully

Here first comparison grid : Clip_L_Text_Encoder_Training_Early_Experiment.jpg

27 August 2024 Update V2

Test prompts improved and moved into a folder

New Best_Configs_Better_Colors folder added

The difference is using Timestep Sampling as shift and using Discrete Flow Shift value as 1

I will do more research about this

You can see quick research comparison grid here : Shift_Value_1_Test.jpg

Shift timestep sampling definitely improved colors and overall composition

In some images you will see reduced likeness but in some images even better

It requires more research

Clip L training arrived (smaller text encoder)

Both Clip-L training and Timestep Shift sampling experiments are being done right now

27 August 2024 Update

Batch size experiments section added and config for 4x GPU added

Configs are inside Best_Configs folder inside above zip file

34 Distinct unique prompts for testing added to the zip file

Hopefully I am going to test time step shift more and Clip L Text Encoder training (Kohya just announced)

Newest Configs :

Rank 1 > Rank 2 > Rank 3 and so on - quality assessment

The quality difference between Rank 1 to Rank 6 are not very much but starting from Rank 7 the quality will degrade since we reduce training resolution and LoRA rank

So you can pick faster over slower if you are in hurry

These are early steps speeds (like first 50 and Kohya doesn't display latest step speed) so as you train more you will get like at least 25% faster speeds

These are raw VRAM usages so you need to look your VRAM usage before starting training

The tests are made on Linux on Massed Compute on RTX A6000 GPU - 1024x1024 resolution - 128 LoRA rank

Rank_1_28700MB_Slow.json - 16bit - 8.53 second / it

Rank_2_27360MB_Fast.json - 16bit - 4.49 second / it

Rank_3_18246MB_Slow.json - 8bit - 8.62 second / it

Rank_4_16960MB_Fast.json - 8bit - 4.61 second / it

Rank_5_11498MB_Slow.json - 8bit - Single Layers - 12.12 second / it

Rank_6_10222MB_Fast.json - 8bit - Single Layers - 9.42 second / it

Rank_7_9502MB.json - 8bit - Single Layers - 8.61 second / it - 896px

Rank_8_15406MB.json - 8bit - 3.71 second / it - 64 LoRA Rank - 896px

Rank_9_7514MB.json - 8bit - Single Layers - 5.8 second / it - 64 LoRA Rank - 512px

My suggestions for GPUs

8 GB GPUs : Rank_9_7514MB.json

10 GB GPUs : Rank_7_9502MB.json

12 GB GPUs : Rank_5_11498MB_Slow.json

16 GB GPUs : Rank_5_11498MB_Slow.json - if you need speed : Rank_8_15406MB.json

24 GB GPUs : Rank_3_18246MB_Slow.json

48 GB GPUs : Rank_1_28700MB_Slow.json

Batch Size Experiments And Multi GPU Usage

I have used Rank_2_27360MB_Fast.json config to test batch size impact on RTX A6000

The speed gain from batch size is almost none thus I don't suggest since you lose quality

Lesser batch size = better quality

Only Batch size 2 gives you some gain so you may use it if you wish

Batch size 1 : 4.54 second / it : effective speed same

Batch size 2 : 7.98 second / it : effective speed per step 3.99 second / it

Batch size 3 : 12.43 second / it : effective speed per step 4.14 second / it

Batch size 4 : 15.28 second / it : effective speed per step 3.82 second / it

Batch size 5 : 20.18 second / it : effective speed per step 4.03 second / it

Therefore I have added batch size 1 but 4 A6000 GPU Speed config : 4x_GPU_Batch_Size_1.json

With this config you get 5.75 second / it and effective speed per step is : 1.4375 second / it

When you use multiple GPU you need to divide epoch count to number of GPUs

So 200 epoch becomes 50 for 4x GPU

Also increase LR with this formula - best LR x (batch size * number of GPUs / 2) so in this case 0.00005 * (4/2) = 0.0001

So if you make batch size 2 it becomes = 0.00005 x (2x4/2) = 0.0002

The zip file now has 4x_GPU_Batch_Size_1.json and 4x_GPU_Batch_Size_2.json

I suggest you to use 4x_GPU_Batch_Size_1.json on a 4x RTX A6000 GPU machine

Inference Config for Evaluating Results of Experiments

I use FP16 FLUX dev model in 16 bit precision with UniPC sampler and 30 steps and SwarmUI grid system

You Don't Necessarily Need Class Prompt With FLUX

In the below grid images

trained with ohwx man : model name > Best_v2-000150 - first column

with only ohwx : model name > Best_v2_only_OHWX-000150 - second column

And then generated images with

prompt having ohwx man > More_Time_Shift_Test_and_Only_Ohwx_Token_Tested_and_Ohwx_Man_Used_in_Prompt.jpg

prompt only have ohwx > More_Time_Shift_Test_and_Only_Ohwx_Token_Tested_and_Only_Ohwx_Word.jpg

prompt only have man > More_Time_Shift_Test_and_Only_Ohwx_Token_Tested_and_Only_Man_Word.jpg

I predict that the T5 Text Encoder already encode image in a way that it fully tokenize it. So even if we don't define captions, our images are fully tokenized

This also results in overtraining in our case

Hopefully I will try lower LR and Clip-L training to solve this issue and I will try fine-tuning as well

So if you are going to train a 2 same class like 2 man in same training you can just caption them as ohwx and bbuk and such and try it

I still find ohwx man slightly better than ohwx

Also full captions reduces likeliness of a person training

Different LoRA Ranks Experiments

Full LoRA Rank impact : https://huggingface.co/MonsterMMORPG/Generative-AI/resolve/main/Full%20LoRA%20Rank%20Impact.jpg

Higher LoRA rank learns more details but overfits model more. I like 128 but it is personal. Look grid above for full impact

Higher Resolution Training Impact

I have done trainings at 1024x1024 (default), 1280x1280 and 1536x1536px

Then I have generated images in 1024x1024, 1280x1280, 1536x1536 and 1920x1080px for each training

Higher resolution training slightly improves quality but not dramatic probably doesn't worth the speed loss it causes

Full grid results are below

High Res Training Comparison 1024px.jpg , High Res Training Comparison 1280px Part 1.jpg , High Res Training Comparison 1536px Part 1.jpg , High Res Training Comparison 1536px Part 2.jpg

QKV Split Attention and JoyCaption Detailed Captioning Trainings Results

I have comprehensively tested QKV split attention and detailed captioned dataset trainings

I don't find split QKV brings benefit and brings huge speed loss

For person training detailed captions reduced the resemblance and didn't improve and flexibility or quality. But the resemblance loss was way lower than SDXL

So my conclusions are do not use split QKV and detailed captions for training a person

For style training probably detailed Joy captions will work great - i should test it too

Full grid results are below

QKV Split and JoyCaption Result 1024px.jpg , QKV Split and JoyCaption Result 1920px.jpg

Time Step Sampling Shift Experiments

This one hugely improved stylization capability of the model but reduced resemblance and the realism

I plan to hopefully do more research on this - 8 more trainings

Wait for them. Full grids below

Time_Step_Sampling_Shift_Grid_Full_Grid.jpg

Apply T5 Attention Mask and detailed Regularization / Classifications Images Impact Experiments

I find that Apply T5 Attention Mask improves quality but the tradeoff is slightly improved VRAM and some serious speed los

I also have compared my training workflow with default CivitAI FLUX LoRA training and ours much better

Regularization / classification images didn't improve results with none of the tested configs - I tested so many - check grids to see in full detai

Full grids are shown below

Reg_T5_Attention_Mask_CivitAI_50_Epoch.jpg, Reg_T5_Attention_Mask_CivitAI_100_Epoch.jpg, Reg_T5_Attention_Mask_CivitAI_150_Epoch.jpg, T5_Attention_Mask_v1.jpg, Civit_Reg_15GB_150_Epoch_Compare.jpg

More Different Experiments Are Historically shared below read the entire thread

-

24 August 2024 Update V2

Windows_Download_Training_Model_Files.bat file added to download necessary training model files into the bat file run directory

A 1-min video published to show how to set Accelerate for Kohya SS GUI : https://youtu.be/adVhm9aI9Gc

This set fixes caching stuck problem

24 August 2024 Update

Massed Compute installer fixed and super simplified just 1 step and auto opens browser

Massed compute instructions updated and auto models downloader added - downloads all necessary models automatically Dev FP16, T5 FP16, VAE and Clip

Fix_For_FLUX_Step_2.bat improved - don't forget to run this for lower VRAM usage and 10GB Config

10 GB config updated for latest GUI

Solution for KeyError: 'time_embed.0.weight' error

When you loaded the config make sure that this is selected

23 August 2024 Update

Massed Compute and RunPod Installers fully added (read Massed_Compute_Kohya_FLUX_Instructions and RunPod_Install_Instructions)

16_GB_Config_May_Be_Lower_Quality_In_Test added - still in test - changes are rank 32 and 896 pixel resolution training but super fast compared to lowest VRAM

Download newest Kohya_GUI_Flux_Installer_v13.zip and extract into any folder

48_GB_GPUs_v2 (27 GB VRAM) and 24_GB_GPUs_v2 (17 GB VRAM) yields almost same quality

10_12_16GB_GPUs_v2.json (yields almost same quality as best ones but 3-5 times slower due to VRAM optimizations) - uses 10183 MB peak so if uses shared VRAM and be too slow reduce LoRA Rank or resolution to like 896x896 - I am testing LoRA rank impact at the moment

16_GB_Config_Slightly_Lower_Than_10GB_But_5_Times_Faster.json yields lower quality than 10_12_16GB_GPUs_v2.json but 3-5 times faster than it

Newest test Grids are as below

Reg_T5_Attention_Mask_CivitAI_50_Epoch.jpg, Reg_T5_Attention_Mask_CivitAI_100_Epoch.jpg, Reg_T5_Attention_Mask_CivitAI_150_Epoch.jpg, T5_Attention_Mask_v1.jpg, Civit_Reg_15GB_150_Epoch_Compare.jpg

Tested all configs file updated to V5 : Tested_All_Configs_V6.zip

So what are the newest changes? I have enabled T5 Attention Mask which I believe slightly improves results

T5 Attention Mask Tradeoffs 

However T5 Attention Mask increases VRAM usage by 1 GB for 16gb, 24gb and 48 gb configs, thus I didn't enable it for 16 GB config. Now 48 GB config uses 28 GB, 24 GB config uses 18 GB with attention masking

Moreover, T5 Attention Mask slows down the training significantly so that is another tradeoff

What does it do explained here : https://poe.com/s/EHIviAdVuZds5XBGDYWP

What is next? I am going to test quality impact of different LoRA network ranks next 4, 8, 16, 32, 64, 128

16 GB Config yielded slightly worse results than 10 GB config probably due to reduced resolution and LoRA network rank

CivitAI default settings training performed worse than our very best config with exactly same dataset and caption

Regularization / classification images tests failed. Still yields non-resemblance images. I don't see its benefit at this point. Tested with different Prior loss weight parameters but non yielded better results

Windows training is still significantly slower than Linux training. I am still researching a fix for this issue. Asked on Pytorch and Diffusers GitHubs

Full Kohya SS GUI setup screenshots added to the installer zip file

22 August 2024 Update

How to setup Kohya interface for FLUX full screenshot : Example Full Setup.jpg

FLUX LoRA training almost perfected

Use LoRA tab to load configs not DreamBooth

I also made 1 click Kohya SS GUI installer for FLUX

Run Windows_Install_Step_1.bat - select option 1 and then exit with option 7 once completed - no need other options

Run Fix_For_FLUX_Step_2.bat and then use Windows_Start_Kohya_SS.bat to start

Step 2 will improve training speed like 100% and VRAM usage like 25% with just proper library upgrade

Same upgrade added for Massed Compute read the

You are ready to use FLUX training with most performant way now

From newest configs I feel like Best_v1_5e_5_max_grad_norm_0 is the best

But Best_v1_5e_5 is also very good. There is like some lightning difference

When you extract the above zip file you will see configs as

48_GB_GPUs.json for 48 GB GPUs

24_GB_GPUs.json for 24 GB GPUs - uses around 17-18 gb with newest libraries

16GB_Compare_Speed_With_10_12_16_Config.json - If you have 16 GB try this and compare speed with 10_12_16GB_GPUs.json

10_12_16GB_GPUs.json - For 10, 12, 16 GB GPUs. If you have 10 GB and be very slow you can reduce LoRA Rank and also resolution to 512,512

512px training yields inferior quality and VRAM usage almost same - but 2.5x speed

New tested configs full quality grids : 50 epoch (750 steps) , 100 epoch (1500 steps) , 150 epoch (2250 steps)

For 24 GB Config RTX 3090 Speed should be close to 4 second / it - 1024x1024px

For 10_12_16 GB config speed drops like 3 to 5 times due to full optimizations

Training dataset 15 images, 1 repeat, train like 200 epochs

22 August 2024 Update

Section "FLUX Huge Sampler + Scheduler Test For a Very Hard Prompt" added to the post

Section "FLUX Guidance Scale Grid Test on LoRA" added to the post

Newests tests completed and new best configs uploaded as

Surprise that now wen have 10 GB config (LoRA rank 128) - 10 GB config will be published hopefully in 8-9 hours

If you have 10 GB GPU try to reduce LoRA rank until it works better also minimize your VRAM usage before starting the training

So the current best configs are like this

For 48 GB GPU : 6e_05_best_raw_sigmoid.json

For 24 GB GPU : 6e_05_best_raw_sigmoid_24GB.json,

For 10, 12, 16 GB GPUs : lowest_vram.json

For lowest VRAM config to work you have to activate VENV of Kohya GUI and execute below commands 1 by 1

This below thing will speed up training almost 100% and reduce VRAM usage greatly but results unknown yet (training atm). You can also do that if you have 24 GB GPUs like 4090

First update Kohya to latest, run 1 time, then activate venv and install below, then edit gui.bat file and comment or remove python.exe .\setup\validate_requirements.py and if %errorlevel% neq 0 exit /b %errorlevel% to prevent auto reinstall requirements

pip install torch==2.4.0+cu124 --index-url https://download.pytorch.org/whl/cu124

pip install torchvision==0.19.0+cu124 --index-url https://download.pytorch.org/whl/cu124

21 August 2024 Update

So far 16 different tests completed

For 48GB GPUs use 6e_05_bf16_128_rank_full_bf16.json - train up to 200 epochs and compare checkpoints

For 24GB GPUs use 6e_05_fp8_bf16_accelerate_full_bf16_32_rank.json - train up to 200 epochs and compare

I will try to get better results and reduce number of necessary epochs

I will like reg images would help but I haven't tested yet

20 August 2024 Update

So far tested configs added to the attachment and more results added to the post

More configs are on training

Windows Requirements

Python 3.10, FFmpeg, Cuda 11.8, C++ tools and Git

If it doesn't work make sure to below tutorial and install everything exactly as shown in this below tutorial

https://youtu.be/-NjNy7afOQ0

How To Install

I am going to use GUI version of the Kohya

https://github.com/bmaltais/kohya_ss

Currently It doesn't have FLUX at the main branch

So clone it into a new folder like c:/flux_train

Before installing open a cmd

Type : git checkout sd3-flux.1

Then install as usual

FLUX Training Discussions With Lots of Info

https://github.com/bmaltais/kohya_ss/issues/2701

https://github.com/kohya-ss/sd-scripts/pull/1374

Models Links Downloads

FLUX dev FP16 (23.8 GB) : https://huggingface.co/OwlMaster/realgg/resolve/main/flux1-dev.safetensors

Download Clip L (250 MB) : https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors

T5 XXL FP16 (9.8 GB) : https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/t5xxl_fp16.safetensors

FLUX VAE (335 MB) : https://huggingface.co/OwlMaster/realgg/resolve/main/ae.safetensors

FLUX Fine Tuning Lower VRAM Optimizations

--optimizer_type adafactor --optimizer_args "relative_step=False" "scale_parameter=False" "warmup_init=False"

--timestep_sampling sigmoid --model_prediction_type raw --guidance_scale 1.0

--blockwise_fused_optimizers --double_blocks_to_swap 6 --cpu_offload_checkpointing

Options are almost the same as LoRA training. The difference is `--blockwise_fused_optimizers`, `--double_blocks_to_swap` and `--cpu_offload_checkpointing`. `--single_blocks_to_swap` is also available.

`--blockwise_fused_optimizers` enables the fusing of the optimizer for each block. This is similar to `--fused_backward_pass`. Any optimizer can be used, but Adafactor is recommended for memory efficiency. `--fused_optimizer_groups` is deprecated due to the addition of this option for FLUX.1 training.

`--double_blocks_to_swap` and `--single_blocks_to_swap` are the number of double blocks and single blocks to swap. The default is None (no swap). These options must be combined with `--blockwise_fused_optimizers`.

`--cpu_offload_checkpointing` is to offload the gradient checkpointing to CPU. This reduces about 2GB of VRAM usage.

--network_args train_blocks=single - reduces vram

All these options are experimental and may change in the future.

The increasing the number of blocks to swap may reduce the memory usage, but the training speed will be slower. `--cpu_offload_checkpointing` also slows down the training.

Swap 6 double blocks and use cpu offload checkpointing may be a good starting point. Please try different settings according to VRAM usage and training speed.

Previous Master Kohya Tutorials

How To Do SDXL LoRA Training On RunPod With Kohya SS GUI Trainer & Use LoRAs With Automatic1111 UI

Become A Master Of SDXL Training With Kohya SS LoRAs - Combine Power Of Automatic1111 & SDXL LoRAs

Massed Compute

Massed compute A100 discount : A100_2ybG1e9StAdA

Auxiliary Scripts and Tutorials

Learn how to use SwarmUI : https://youtu.be/HKX8_F1Er_w

Learn how to use SwarmUI on Cloud (RunPod, Massed Compute, Kaggle) : https://youtu.be/XFUZof6Skkw

Learn how to use FLUX with SwarmUI : https://youtu.be/bupRePUOA18

Learn how to download models from Hugging Face and CivitAI and how to upload Hugging Face for backup : https://youtu.be/X5WVZ0NMaTg

FLUX Guidance Scale Grid Test on LoRA

As you know FLUX has CFG set as 1

FLUX dev model has FLUX guidance scale

You can download full grid test from this link and check out

I feel like FLUX Guidance 4.0 is best but you can push more to get more accurate prompt following

FLUX Huge Sampler + Scheduler Test For a Very Hard Prompt

Prompt is : A cinematic shot featuring a photo-realistic image of an Ohwx man riding an enormous and terrifying Tyrannosaurus rex in the heart of Jurassic Park. The scene is set against a backdrop of dense, prehistoric jungle with towering trees and lush greenery, creating an intense and thrilling atmosphere. The T-rex is depicted in all its majestic and fearsome glory, with its massive jaws open wide, sharp teeth glistening, and powerful muscles rippling as it charges forward. The Ohwx man, in contrast, is shown with a determined and fearless expression as he expertly rides the gigantic predator, his posture exuding confidence and control. The image is captured in a highly cinematic style, with dramatic lighting and dynamic angles that emphasize the scale and intensity of the scene. The focus is partially directed towards the face of the Ohwx man, ensuring that his features are clearly visible and convey a sense of resolve and bravery, while maintaining the overall composition of the dynamic action scene.<segment:face,0.7>photo of ohwx man

So far my findings are as below as promising sampler + scheduler

Full grid link : click here

euler + normal - default

euler + karras - cinematic look

heun + karras , heunpp2 + karras

dpm_2 + karras or exponential

dpm_fast + normal

lcm + normal or karras : cartoon look

unipc + normal or karras

I did bigger tests and my conclusion is that sadly Karras scheduler which makes it more realistic is only useable with DMP_2 and even that is not great

Also I think best sampler is UniPC

Below all test results

LCM_Normal_vs_Karras.jpg , Eular_A_Normal_vs_UniPC_Normal.jpg , Eular_Normal_vs_Karras.jpg , UniPC_Normal_vs_Karras.jpg , DPM_2_Normal_vs_Karras.jpg

Experiments and VRAM Usages

A6000

Accelerate launch Mixed precision BF16 - highvram - 128 rank : 44.18 GB - 8.25 second / it

Accelerate launch Mixed precision BF16 - full BF16 - highvram 128 rank : 42.58 GB - 8.25 second / it

Accelerate launch Mixed precision BF16 - lowvram - 128 rank : 44.18 GB - 8.25 second / it : 44.18 GB - 8.25 second / it

Accelerate launch Mixed precision BF16 - full BF16 - lowvram 128 rank : 42.58 GB - 8.25 second / it

Accelerate launch Mixed precision FP16 - highvram - 128 rank : 44.14 GB - 7.83 second / it

Accelerate launch Mixed precision FP16 - full FP16 - highvram - 128 rank : 42.55 GB - 7.83 second / it

Accelerate launch Mixed precision BF16 - highvram - 128 rank - Memory efficient attention : 44.18 GB - 8.25 second / it

Accelerate launch Mixed precision BF16 - highvram - 128 rank - Split Mode : error

Accelerate launch Mixed precision BF16 - highvram - 128 rank - Split Mode train blocks double : error

Accelerate launch Mixed precision BF16 - highvram - 128 rank - Split Mode train blocks single : error

Accelerate launch Mixed precision FP16 - fp8 base - highvram - 128 rank : 33.53 GB - 7.99 second / it

Accelerate launch Mixed precision FP8 - fp8 base - highvram - 128 rank : error

Accelerate launch Mixed precision FP8 - fp8 base - full FP16 - highvram - 128 rank : error

Accelerate launch Mixed precision FP16 - fp8 base - highvram - full FP16 - 128 rank : 24.39 GB - 7.96 second / it

Accelerate launch Mixed precision BF16 - fp8 base - highvram - 128 rank : 33.55 GB - 8.38 second / it

Accelerate launch Mixed precision BF16 - fp8 base - full BF16 - highvram - 128 rank : 24.4 GB - 8.35 second / it

Accelerate launch Mixed precision FP16 - fp8 base - highvram - full FP16 --network_args train_blocks=single - 128 rank : 18.39 GB - 18 second / it

Accelerate launch Mixed precision BF16 - fp8 base - highvram - full BF16 --network_args train_blocks=single - 128 rank : 18.39 GB - 21 second / it

Running Experiments

1e_04_bf16_128_rank : completed - 48 gb GPU

1e_04_fp16_128_rank : completed - 48 gb GPU

1e_04_bf16_128_rank_full_bf16 : completed - 48 gb GPU

1e_04_fp16_128_rank_full_fp16 : completed - 48 gb GPU

1e_04_fp8_32_rank : completed - 23.58 GB

1e_04_fp8_full_fp16_32_rank : completed - 23.28 GB

1e_04_fp8_bf16_accelerate_32_rank : completed - 23.57 GB

1e_04_fp8_bf16_accelerate_full_bf16_32_rank : completed - 23.28 GB

Early Testing Results

Looks like FP16 accelerator training yields way inferior results compared to BF16 accelerator training

This results is very early experiments only 50 epoch

More results obtained

So far best working cases are 1e_04_bf16_128_rank_full_bf16 and 1e_04_fp8_bf16_accelerate_full_bf16_32_rank

FP16 training broken

Starting 8 more test right now

These below images has no face inpainting raw output

These are early results of my extensive FLUX LoRA training but 8 bit is way inferior to 16 bit atm

8 bit looks undertrained perhaps and 16 bit is overtrained with current LR and 150 epoch

Gonna do more testing now

No face inpainting raw 1024x1024 images

 Some more full comparisons attached to attachments

8 bit BF16 vs full BF16 Comparison.png

16 bit BF16 vs full BF16 Comparison.png

8 bit vs 16 bit full Comparison.png

New Started Trainings

48 GB Configs - uses 42.6 GB VRAM

9e_05_bf16_128_rank_full_bf16 - completed

8e_05_bf16_128_rank_full_bf16 - completed

7e_05_bf16_128_rank_full_bf16 - completed best at 150 epoch

6e_05_bf16_128_rank_full_bf16 - completed very best at 200 epoch

24 GB Configs - uses 23.30 GB VRAM

2e_04_fp8_bf16_accelerate_full_bf16_32_rank

3e_04_fp8_bf16_accelerate_full_bf16_32_rank

9e_05_fp8_bf16_accelerate_full_bf16_32_rank

8e_05_fp8_bf16_accelerate_full_bf16_32_rank

New Training Results for 48 GB Configs

So far I have tested 48 GB configs results and very best is 6e_05_bf16_128_rank_full_bf16 however it requires 200 epoch training for 15 images

At 150 epochs 7e_05_bf16_128_rank_full_bf16 yields best results but it is not as good as 200 epoch of 6e_05_bf16_128_rank_full_bf16

New grids uploaded to attachments , 50_epoch_LowerLR_48_tests.jpg , 100_epoch_LowerLR_48_tests.jpg , 150_epoch_LowerLR_48_tests.jpg , 200_epoch_LowerLR_48_tests.jpg, Face_Inpainted_48_GB_Best_Comparison.jpg

Tested configs updated to V2 - look attachments for the zip file

Here some examples

24 GB Experiments

I feel like same LR of 48 GB works

So I decided to go with single version tests from now on

At the very top of the model I added best configs

Also configs zip file updated

24 GB experiments files : 50_epoch_LowerLR_24_GB_config_tests.jpg , 100_epoch_LowerLR_24_GB_config_tests.jpg , 150_epoch_LowerLR_24_GB_config_tests.jpg , 200_epoch_LowerLR_24_GB_config_tests.jpg

Started 7 New Trainings

6e_05_best_with_reg_images - completed results below

6e_05_best_max_grad_norm_0 - completed results below

6e_05_best_raw_sigmoid - completed results below

6e_05_best_raw_sigma - completed results below

6e_05_best_additive_sigma - completed results below

6e_05_best_sigma_scaled_sigma - completed results below

6e_05_best_additive_uniform - completed results below

Prediction_Type_Timestep_Sampling_200_Epoch.jpg , Prediction_Type_Timestep_Sampling_150_Epoch.jpg