Home Creators Posts Import Register Favorites Logout

Downloads

Content

Kohya SS GUI FLUX LoRA training step by step, with the best optimized configs for 4 GB to 48 GB GPUs and a 68 minute Windows tutorial with 74 chapters.

Runs on local Windows and in the cloud on RunPod, Massed Compute and SimplePod, with a cloud tutorial for multi-GPU training. V41 fixes FLUX training bugs and updates the SDXL DreamBooth / fine-tuning configs; FLUX Dev, SRPO and Krea LoRA comparison grids are attached.

Full step by step Kohya SS GUI FLUX LoRA training. Includes 4 to 48 GB GPUs very best optimized configs. For Windows, RunPod, Massed Compute

Patreon exclusive posts index to find our scripts easily, Patreon scripts updates history to see which updates arrived to which scripts and amazing Patreon special generative scripts list that you can use in any of your task.

Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord

Please also Star, Watch and Fork our Stable Diffusion & Generative AI  GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)

=======

Windows Tutorial Published - 68 Min - 74 Video Chapters

Tutorials & Resources

20 January 2026 V41

  • Few bugs for FLUX training fixed

  • SDXL DreamBooth / Fine Tuning training configs fully updated after new tests

  • It is 100% recommended to use regularization / classification images with SDXL training, do 1 repeat and use Save Every N Steps method

  • Watch this tutorial : https://youtu.be/EEV8RPohsbw

13 January 2026 V40

  • I have fixed FLUX LoRA training with text encoder

    • ComfyUI will still show some key errors but they are inaccurately displayed and fully working i tested and checked

  • Some SDXL configs added and more will be added and shared later hopefully with more info

  • RunPod template link updated and now we fully support SimplePod which is much faster and cheaper than RunPod

  • SIMPLEPOD CHEAPER AND FASTER THAN RUNPOD

  • As usual follow Massed_Compute_Kohya_FLUX_Instructions.txt and RunPod_SimplePod_Kohya_Instructions.txt to install and use and watch the tutorials

29 November 2025 Update v38

  • Fused Backward Pass enabled back in big VRAM configs since disabling it breaking the training

  • Installers upgraded to uv therefore it is like 100 times faster on RunPod and 10x faster on Windows and Massed compute to install

25 November 2025 Update v35

New tutorial for v35+ published : https://youtu.be/RQHmyJVOHXo

  • I have forked the famous Kohya SS scripts and now I am developing it

  • All of the configs are updated based on new Torch Compile feature and now multiple GPU Fine Tuning / DreamBooth works on 80 GB GPUs perfect

    • Now we have faster 80 GB GPUs configs as well with Torch Compile

  • We have added so many new amazing features to both our SECourses Premium Kohya SS GUI and SECourses Premium Kohya SD Scripts

    • Now our app and GUI supports Torch-Compile

    • So far tested on FLUX DreamBooth / Fine Tuning and LoRA Training

    • Brings performance between 5-20% speed gain depending on configuration with 0 trade-off

    • No quality loss or no extra VRAM usage

    • Works with Block Swapping as well

    • I have added this new feature solely and took approach from famous Kohya Musubi Tuner repo

    • I have added CPU based text encoder caching for lower than 10 GB GPUs - this was not existing in SD Scripts and our new configs are now based on this

  • Moreover, now the GUI has Open All Sections and Close All Sections to quickly open and close all sections, then you can do Ctrl+F to quickly search and find what you want

  • New FP8_Scaled feature added to LoRA training just as in Kohya Musubi Tuner

    • This reduces VRAM usage significantly like from 29.3 GB to 21.6 GB with almost no quality loss, base model is dynamically converted into FP8_Scaled while loading

  • I have added new memory-efficient loading so now it should use lesser RAM compared to before, very good improvement for low RAM machines

  • Also I have fixed a bug in Kohya scripts:

    • Now you will see actual training speed right after very first step if your all images are same resolution

    • If you have different aspect ratio and different resolutions in dataset, you will see actual speed after first epoch not like after 100s of steps

  • Make a fresh install into a new folder, Massed Compute and RunPod fresh installs will install latest version

New Configs With V35

The difference of For_RunPod folder is that it enables more RAM using faster model loading plus paths are set for /workspace automatically

29 October 2025 Update v32

  • I have added a new amazing tool called as Image Preprocessing

  • This tool is extremely important and useful when you do training with bucketing enabled

  • I recommend to use this tool, preprocess your training images and checkout how your images actually used during training

  • Just run Windows_Install_or_Update_Kohya.bat to update

2 October 2025 Update

  • Sadly Bmaltais stopped developing Kohya GUI therefore I forked his repo and now we are going to use myself developed

    • One advantage of this that now we are going to use always latest version of SD Scripts of Kohya

  • I have extremely optimized and significantly improved the installation and therefore now it will be way faster and more accurately installed on Windows, RunPod and Massed Compute

  • I have updated libraries to Torch 2.8, CUDA 12.9, Accelerate 0.48, xFormers 0.33, Flash Attention 2.8.3, Sage Attention 2.2 and Triton 3.4 on all platforms

    • Now it supports all of the GPUs starting from RTX 1000 series to 5000 series + cloud GPUs like RTX A6000, A100, H200, B200 etc

  • Moreover I made the app to auto recognize FLUX Krea Dev and FLUX SRPO models as FLUX.1 - remember you have to enable that checkbox

    • All you need to do is after loading the config, select downloaded FLUX SRPO as a base model not FLUX Dev model in model path

  • I have trained the new FLUX SRPO model with our existing DreamBooth configs and compared it to FLUX Krea and FLUX Dev base model

  • I can confidently say that the FLUX SRPO model is perfectly trainable with our config and it is a little bit more realistic than FLUX Dev

    • So for realism from now on I recommend FLUX SRPO

  • Here below base 1024x1024 no face restoration or upscale made results below

  • FLUX SRPO is an extremely realistic base model compared to FLUX Dev - it is a special fine tune : https://github.com/Tencent-Hunyuan/SRPO

  • I recommend to get latest zip file and make a fresh install into a new folder if you want to upgrade to the latest version since a lot of installation process changed

  • The model downloader script upgraded to our special ultra FAST and robust model downloader - like uGet with 16 connections + SHA 256 verification

  • The Windows_Download_Training_Model_Files.bat will ask you which model you want to download

  • On RunPod and Massed Compute read the instruction txt files and you will see commands to download any of the models directly

Windows Requirements

Massed Compute (Recommend Cloud) :

RunPod (Cloud):


13 August 2025 Update

  • I have trained FLUX Krea Dev model with our FLUX Dev LoRA configs and compared the results - inside LoRA_Tab_LoRA_Training_Best_FLUX_Configs folder

    • Our model downloader in zip file now auto downloads FLUX Krea Dev too

    • So after loading your config just change base model to FLUX Krea Dev

    • FLUX Krea Dev Tutorial here

      • https://youtu.be/8MvvuX4YPeo

        • 15:31 FLUX Krea Dev vs FLUX Dev: A Detailed Side-by-Side Image Comparison

        • 16:26 How to Easily Train Your Own LoRAs on the New FLUX Krea Dev Model

        • 17:02 Complete Workflow for Generating High-Quality Images with FLUX Krea Dev

        • 18:20 The Final Verdict: Side-by-Side Result of FLUX Krea Dev vs FLUX Dev

  • I feel like FLUX Krea Dev needs a little bit higher learning rate or longer training

    • I recommend longer training

  • You can see full size grid comparisons below - trained on 28_imgs_dataset.png

  • I may also research Chroma model and publish presets for it, currently my focus is Qwen Image which I believe will be better than FLUX Dev in every aspect

  • Hopefully full tutorial and very easy to use workflows and presets coming soon for Qwen Image model training i am working on Gradio App and presets

13 July 2025 Update

  • Gradio broken thus added temporary fix : Temp_Fix_Gradio_Error.bat

  • RunPod and Massed Compute fix auto applied

29 May 2025 Update

  • 32 GB RAM configs added - not VRAM system RAM

    • They are inside LoRA_Tab_LoRA_Training_Best_FLUX_Configs inside 32 GB RAM Configs - Not VRAM - RAM folder

    • The difference is that you have to use flux1-dev-fp8.safetensors and now the config has enabled fp8 base unet

  • Windows_Download_Training_Model_Files.bat updated to prevent possible errors during download of models

13 May 2025 Update

  • Now on RunPod and Massed Compute our installer supports RTX 5000 series as well as older GPUs like RTX 3090, RTX 4090 etc

  • Upgraded to Torch 2.7 and CUDA 12.8

4 May 2025 Update

  • First run installer and then run Windows_RTX5000_Series_Upgrade_Run_After_Install_Finished.bat

    • Now it uses official Torch 2.7, CUDA 12.8, and myself compiled xFormers

    • This is required for all GPUs

  • Training models uploaded to myself hosted XET enabled repo for even faster and more stable downloads : https://huggingface.co/MonsterMMORPG/Kohya_Train/tree/main

  • All configs are up-to-date with best settings

  • Amazing 22 special prompts added for woman trainings testing into Test_Prompts folder

17 November 2024 Update

  • Huge improvements arrived with newest block swapping feature of Kohya

  • Model downloaders are updated and made super fast compared to before on all platforms like Windows, RunPod and Massed Compute - up to 10 times faster

    • On Massed Compute downloading all training models only took 1 minute (over 30 GB)

  • All configs are updated and please look at the 0_Configs_Explanation_Must_Read.jpg inside the LoRA_Tab_LoRA_Training_Best_FLUX_Configs folder

  • Pick the config depending on your GPU, the quality you target and the speed you need

  • Please watch above listed tutorials to fully learn how to use

  • Update your Kohya to latest via Windows_Install_Torch_2_5_Dev_Huge_Speed_Up.bat or it is better to reinstall Kohya make a fresh install

31 October 2024 Update

  • xFormers and Torch 2.5.1 fully officially published

  • Thus use Windows_Install_Torch_2_5_Dev_Huge_Speed_Up.bat file

  • Massed Compute and RunPod installers also updated for Torch 2.5.1 and xFormers 0.0.28.post3

  • All configs both Fine-Tuning / DreamBooth and LoRA updated to xFormers instead of SDPA

    • I find that xFormers slightly yields better results

  • Recommend RunPod template changed to below

    • RunPod Pytorch 2.2.0

      • runpod/pytorch:2.2.0-py3.10-cuda12.1.1-devel-ubuntu22.04

7 October 2024 Update

  • New amazing prompts added inside New_Test_Prompts folder

4 October 2024 Update

  • Installer files updated according to the latest Bmaltais Updates

  • xFormers is kept with 0.0.28.post1 unless you install Torch 2.5 for FLUX

  • Still no xFormers available for Torch 2.5 yet

  • Bmaltais added a new option to bat file so we won't overwrite anymore gui.bat file - new command is : --noverify

  • When you use Windows_Update_Kohya_and_Fix_FLUX_Step2.bat on existing installation you will get error, either do a fresh install or open a cmd on kohya_ss folder and execute git stash and then run uıpdater

  • Bat files are renamed to better like below with order

    • 1: Windows_Install_Step_1.bat

    • 2: Windows_Update_Kohya_and_Fix_FLUX_Step2.bat

    • 3: Windows_Install_Torch_2_5_Dev_Huge_Speed_Up.bat

    • To start 4: Windows_Start_Kohya_SS.bat

19 September 2024 Update

  • Famous 1-layer training configs added

  • There are 19 double blocks, 38 single blocks in the FLUX model

  • I have tested 8 blocks from double blocks and 16 blocks from single blocks and decided that single block 7 is best

  • However, the quality is lower than full LoRA training

  • Only advantage is generated file size

  • I have shared my full thoughts, comparisons, research results and conclusions here: Link will be added once published

  • New 1-layer training configs are inside Single_Layer_Configs folder

  • Quality 1 is better than Quality 2 and so on, so best quality is Quality 1 config

  • Quality_1_25800MB_3_7_Second_IT.json means that it is using min 25800 MB VRAM and per step speed is 3.7 second on RTX A6000 (almost same as RTX 3090)

14 September 2024 Update

9 September 2024 Update

  • I have tested training T5 XXL text encoder with 7 new unique trainings

  • The results are not a clear improvement

  • Check the below grids to see the results

  • T5-Training-Experiments-Prompt-Set-1-Full-Grid.jpg , T5-Training-Experiments-Prompt-Set-2-Full-Grid.jpg , T5-Training-Experiments-Raw-vs-T5-Captioned-Full-Grid.jpg

  • With T5 XXL, with same LR, it almost adds nothing new

  • With T5 XXL, with reduced unet LR, we may get somewhat better results but it is subjective and also the likeliness in some images gets reduced - so you need to generate more images to get perfect likeliness

  • I tested T5 XXL training with captions (Joycaption used) as well but still I didn't see any reason or improvements

  • All results are on above grids

  • One of our followers said that T5 XXL helps when you train something that has text on it

  • So if you are training something with text, you can try

  • New testing prompts added to the Test_Prompts folder

  • I started using face_yolov9c.pt - generate yolov8 folder inside SwarmUI Models folder and put there

  • So I have included the new configs and latest configs are as below

  • Rank 3 - T5 would barely fit on a 24GB GPU so make sure to lower your VRAM usage a lot

Ultra Detailed Research and Development Article

If Your Training Terminating at the Stage of Caching Latents

Automatic Installers and Configs

  • Download attached Kohya_GUI_Flux_Installer zip file

  • Latest version will be on the top of the post and in the attachments

  • Extract into a main drive like c:/Kohya_GUI_Flux_Installer_v17 or d:/Kohya_GUI_Flux_Installer_v17

  • Don't put into c:/windows c:/users etc

  • Currently Kohya SS GUI main branch doesn't have FLUX so we are going to install sd3-flux.1 branch

  • Use Windows_Install_Step_1.bat file

  • Select option 1

  • Once it is completed setup accelerator as shown here (56 seconds video) : https://youtu.be/adVhm9aI9Gc

  • Then close and do not install any other options

  • After this step, for updating installation to latest version and fix libraries run Update_Kohya_and_Fix_FLUX_Step2.bat file

  • Whenever you want to update Kohya SS GUI to latest version use this file

  • You can run this file every time before starting a new training to get latest fixes and changes

  • Once FLUX arrives to main repo I will update installer bat files

  • After these steps use Windows_Download_Training_Model_Files.bat file

  • It will download training necessary files into the same folder

  • These files are from below links

  • Models Links Downloads

  • Do not use other files that you have because you may get error due to incompatibility

  • FP8 version base support added but I didn't train with it so can't tell if working as expected or not. Using FP16 base model will not cause more VRAM usage. It will be automatically casted into accurate precision. I prefer and suggest FP16 base model.

How To Use Config Files

  • After started Kohya SS GUI, pick the config according to the below comparison chart from Best_Configs folder or Best_Configs_Better_Colors (more info below)

  • Set your training dataset folders, output folder, output file name, number of epochs, save every n-epochs checkpoint, training model files paths and you are set

  • By default the configs will train 200 epochs and save every 25 checkpoints (you can also save like every 10 checkpoint)

  • But if you have too many images like 50 or 100 you can set lower epoch count as well - still if you have time train more and save more frequent checkpoints

  • We don't use regularization / classification images with FLUX because it doesn't improve results

  • Comparisons posted on research article which linked at the top

  • Thus we use repeating 1 - very important

Best_Configs_Better_Colors Folder

  • The difference of this folder that it uses Timestep Sampling as Shift and using Discrete Flow Shift value as 1

  • You can see quick research comparison grid here : Shift_Value_1_Test.jpg

  • An imgsli comparison (10 different images) here (a is Sigmoid b is Shift sampler) : https://imgsli.com/MjkxNzAy 

  • Shift timestep sampling definitely improved colors and overall composition

  • In some images you will see reduced likeness but in some images even better

  • It requires more research

Batch Size Experiments And Multi GPU Usage

  • I have used Rank_2_27360MB_Fast.json config to test batch size impact on RTX A6000

  • The speed gain from batch size is almost none thus I don't suggest since you lose quality

  • Lesser batch size = better quality - i tested this so many times and people doesn't properly test

  • You should use more batch size only when you need more speed

  • Only Batch size 2 gives you some gain so you may use it if you wish

  • Batch size 1 : 4.54 second / it : effective speed same

  • Batch size 2 : 7.98 second / it : effective speed per step 3.99 second / it

  • Batch size 3 : 12.43 second / it : effective speed per step 4.14 second / it

  • Batch size 4 : 15.28 second / it : effective speed per step 3.82 second / it

  • Batch size 5 : 20.18 second / it : effective speed per step 4.03 second / it

  • Therefore I have added batch size 1 but 4 A6000 GPU Speed config : 4x_GPU_Batch_Size_1.json

  • With this config you get 5.75 second / it and effective speed per step is : 1.4375 second / it

  • When you use multiple GPU you need to divide epoch count to number of GPUs

  • So 200 epoch becomes 50 for 4x GPU

  • Also increase LR with this formula - best LR x (batch size  number of GPUs / 2) so in this case 0.00005  (4/2) = 0.0001

  • So if you make batch size 2 it becomes = 0.00005 x (2x4/2) = 0.0002

  • The zip file now has 4x_GPU_Batch_Size_1.json and 4x_GPU_Batch_Size_2.json

  • I suggest you to use 4x_GPU_Batch_Size_1.json on a 4x RTX A6000 GPU machine

How To Prepare Dataset

  • I have done extensive testing as shared in above research article

  • I find that ohwx man yields best results when training a person

  • If you are gonna train multiple person you can omit class prompt man and just train as ohwx, bbuk, and such random weird words

  • Since FLUX has an internal Text Encoder alike system it will still have internal captioning effect and learn fully

  • For style training captioning may still yield better results

  • I suggest you to use fully multi-GPU supporting and batch size having with lots of features having JoyCaption app : https://www.patreon.com/posts/joycaption-image-110613301

  • To batch edit generated captions like replacing words, injecting words, use our self developed amazing batch caption editor Gradio APP : https://www.patreon.com/posts/108992085

  • Hopefully I will do more research on style, object like clothings and multi concept training

  • The below is my used dataset

  • It is a low quality dataset since doesn't have expressions, distant shots, different backgrounds and clothings

  • But still works really really good with FLUX

  • So when you prepare a better dataset you will get better results

  • Make sure that your images have very good lightning and focus

  • Training in higher resolution very slightly yields better results look research thread above

  • Training in lower resolution yields significantly lower quality - minimum images should be 1024x1024

  • I have amazing auto subject zoom, crop and resizing scripts

  • Check this video : https://youtu.be/Fbuyu35TkE4

  • Auto subject zoom + cropping and then resizing scripts shared here : https://www.patreon.com/posts/sota-subject-and-88391247

  • Even though I have very extensively tested reg / class images they don't help. You can see results in research post link shared above

  • You can use .txt for caption tokens or just folder names like 1_ohwx man (so it is equal as having .txt files that contains text ohwx man)

  • Set repeating 1 since we do not use regularization / classification images

  • What does repeating means and how it works explained here by Kohya : https://github.com/kohya-ss/sd-scripts/issues/640

  • Below is my used dataset

How To Use FLUX and LoRAs After Trainings Have Been Completed

How To Train and Use On RunPod and Massed Compute

How To Connect SwarmUI from your PC that is Running on Massed Compute - Preffered Cloudflare Way

First open a terminal and execute below commands to install cloudflared

Then run SwarmUI to update to update it latest, then close the started SwarmUI terminal

Then open a new terminal and execute below commands and it will start SwarmUI and will give you a cloudflare link like (my-pills-sailing-pad-netherlands.trycloudflare .com) that you can connect

  • cd /home/Ubuntu/apps/StableSwarmUI/

  • ./launch-linux.sh --launch_mode none --cloudflared-path cloudflared

How To Save and Download Your Models From Hugging Face - CivitAI

Best SDXL and SD 1.5 Configs

Related posts

All posts in this topic: LoRA and Fine-Tuning Training: Kohya, Musubi Tuner, OneTrainer, AI Toolkit · New to SECourses? Start Here: Best and Most Popular SECourses Posts

Files

FLUX LoRA Training Simplified: From Zero to Hero with Kohya SS GUI (8GB GPU, Windows) Tutorial Guide

Ultimate Kohya GUI FLUX LoRA training tutorial. This tutorial is product of non-stop 9 days research and training. I have trained over 73 FLUX LoRA models and analyzed all to prepare this tutorial video. The research still going on and hopefully the results will be significantly improved and latest configs and findings will be shared. Please watch the tutorial without skipping any part. If you are a beginner user or an expert user, this tutorial covers all for you. 🔗 Full Instructions and Links Written Post (the one used in the tutorial) ⤵️ ▶️ https://www.patreon.com/posts/click-to-open-post-used-in-tutorial-110879657 0:00 Full FLUX LoRA Training Tutorial 3:37 Guide on downloading and extracting Kohya GUI 4:22 System requirements: Python, FFmpeg, CUDA, C++ tools, and Git 5:40 Verifying installations using the command prompt 6:20 Kohya GUI installation process and error-checking 6:59 Setting the Accelerate option in Kohya GUI, with a discussion of choices 7:50 Use of the bat file update to upgrade libraries and scripts 8:42 Speed differences between Torch 2.4.0 and 2.5, particularly on Windows and Linux 9:54 Starting Kohya GUI via the gui.bat or automatic starter file 10:14 Kohya GUI interface and selecting LoRA training mode 10:33 LoRA vs. DreamBooth training, with pros and cons 11:03 Emphasis on extensive research, with over 72 training sessions 11:50 Ongoing research on hyperparameters and future updates 12:30 Selecting configurations based on GPU VRAM size 13:05 Different configurations and their impact on training quality 14:22 "Better colors" configuration for improved image coloring 15:58 Setting the pre-trained model path and links for downloading models 16:42 Significance of training images and potential errors 17:08 Dataset preparation, emphasizing image captioning, cropping, and resizing 17:54 Repeating and regularization images for balanced datasets 18:25 Impact of regularization images and their optional use in FLUX training 19:00 Instance and class prompts and their importance in training 19:58 Setting the destination directory for saving training data 20:26 Preparing training data in Kohya GUI and generated folder structure 21:10 Joy Caption for batch captioning images, with key features 21:52 Joy Caption interface for batch captioning 22:39 Impact of captioning on likeness, with tips for training styles 23:26 Adding an activation token to prompts 23:54 Image caption editor for manual caption editing 24:53 Batch edit options in the caption editor 25:34 Verifying captions for activation token inclusion 26:06 Kohya GUI and copying info to respective fields 27:01 "Train images image" folder path and its relevance 28:10 Setting different repeating numbers for multiple concepts 28:45 Setting the output name for generated checkpoints 29:03 Parameters: epochs, training dataset, and VAE path 29:21 Epochs and recommended numbers based on images 30:11 Training dataset quality, including diversity 31:00 Importance of image focus, sharpness, and lighting 31:42 Saving checkpoints at specific intervals 32:11 Caption file extension option (default: TXT) 33:20 VAE path setting and selecting the appropriate VA.saveTensor file 33:59 Clip large model setting and selecting the appropriate file 34:20 T5 XXL setting and selecting the appropriate file 34:51 Saving and reloading configurations in Kohya GUI 35:36 Ongoing research on clip large training and VRAM usage 36:06 Checking VRAM usage before training and tips to reduce it 37:39 Starting training in Kohya GUI and explanation of messages 38:48 Messages during training: steps, batch size, and regularization factor 39:59 How to set virtual RAM memory to prevent errors 40:34 Checkpoint saving process and their location 41:11 Output directory setting and changing it for specific locations 42:00 Checkpoint size and saving them in FP16 format for smaller files 43:21 Swarm UI for using trained models and its features 44:02 Moving LoRA files to the Swarm UI folder 44:41 Speed up Swarm UI on RTX 4000 series GPUs 45:13 Generating images using FLUX in Swarm UI 46:12 Generating an image without a LoRA using test prompts 46:55 VRAM usage with FLUX and using multiple GPUs for faster generation 47:54 Using LoRAs in Swarm UI and selecting a LoRA 48:27 Generating an image using a LoRA 49:01 Optional in-painting face feature in Swarm UI 49:46 Overfitting in FLUX training and training image quality 51:59 Finding the best checkpoint using the Grid Generator tool 52:55 Grid Generator tool for selecting LoRAs and prompts 53:59 Generating the grid and expected results 56:57 Analyzing grid results in Swarm UI 57:56 Finding the best LoRA checkpoint based on grid results 58:56 Generating images with wildcards in Swarm UI 1:00:05 Save models on Hugging Face with a link to a tutorial 1:00:05 Training SDXL and SD1.5 models using Kohya GUI 1:03:20 Using regularization images for SDXL training 1:05:30 Saving checkpoints during SDXL training 1:06:15 Extracting LoRAs from SDXL models

Comments

daniel mendoza

I applied the configuration profile "Rank_3_18246MB_Slow.json - 8bit - 8.62 second / it", and at the end of the training, and generating test images, I noticed that they all come out with vertical lines. The same thing didn't happen to me when I used the previous profile you had created called "24_GB_GPUs". You also have to consider that I used twice the amount of images. With the rank 3 profile I used 88, when I had previously used 44. In addition, the training with rank 3 was at 1024x1024 and previously it was with 768x768. I would have to perform a new training with the initial 44 images in 768x768 to confirm that it is a profile error, but it should not generate those obnoxious vertical stripes. Check: https://i.postimg.cc/fLr9GSXC/ejemplo-1.png https://i.postimg.cc/vBBn2t1D/ejemplo-2.png

SECourses

how do you generate images with swarmui? resolution doesnt have such impact. previous config was also 1024x1024. check your image generation settings and set cfg to 1 dont forget

daniel mendoza

The two images are made with the same configuration and the same workflow.

SECourses

can you try with swarmui to see if it produces such error? current config very similar to latest config you can compare

daniel mendoza

but it may have nothing to do with the json, and it is simply the dataset. It still seems strange to me that when training at 1024x1024 and 88 images it looks worse than when training at 768x768 and 44 images. I will have to repeat the training with the 44 images at 1024x1024 with the 3rank json to discard, otherwise I will simply use the previous version of json that you had placed (24 gb json). I was reviewing the two json and the only difference is this parameter: apply_t5_attn_mask flux. So I guess the problem is mine and has nothing to do with the json. I understand that this parameter increases ram memory consumption but improves the stability of the training.

SECourses

you are right. apply_t5_attn_mask is improving stability of the training and increases VRAM. let me know final results

Samael1976

Hi Furkan! Today (yesterday it worked) I made a runpod with version 17. I used different configuration files (both 48 and 24) but none of them worked. Second thing (less important) I tried to make it work at home that I have a 2060 with 12gb. It doesn't work at home either, it always goes out of cuda (even though I tried even lower ranks). I can tell you that obviously I tried to set FP16 (and also FP8) since the 2060 doesn't support BF16. But probably Kohya (I'm not sure) has a bug because when I go to see the execution of the script, in the accelerate.exe phase it always reads BF16 and never FP16. If you can test Runpod (maybe with optimizer AdamW) you would do me a huge favor (because today we tried in two and nothing).

SECourses

sadly FP16 doesnt work . last time i tested it didnt train. i tested runpod and it was working . what error you got on runpod can you let me know? i will try again and fix if broken

Samael1976

This is the error: Tests carried out - Both with Adafactor (and all your settings) - Both with AdamW (and AdamW8bit) - Tried setting the TE learning rate to 0 "learning_rate_te": 0, "learning_rate_te1": 0, "learning_rate_te2": 0, FIRST STEP INSTALLATION LOG: https://silverider76.iliadboxos.it:27440/share/IKEz4WK2mkaktgh6/01%20-%20First%20Step%20Installation.txt SECOND STEP INSTALLATION LOG: https://silverider76.iliadboxos.it:27440/share/mRMx-MpQKCliFNiI/02%20-%20Second%20Step%20Installation.txt PRINT TRAINING COMMANDS: https://silverider76.iliadboxos.it:27440/share/Q9-JLRx79qecFpOa/03%20-%20Printed%20Training%20Command.txt ERROR LOG: https://silverider76.iliadboxos.it:27440/share/TVPgGmBheL3sLrgA/04%20-%20Error.txt JSON CONFIGATION FILE: https://silverider76.iliadboxos.it:27440/share/_1x3ROpOB9TqbxA6/Rank_1_28700MB_Slow_AniToon.json

SECourses

you are not using accurate template. use pytorch template written in guide please also your config is inaccurate compare it with my zip file config you will see differences. please use lora tab to load after extracted from zip file. i recorded the video editing to publish

Samael1976

- I'm using the template: RunPod Pytorch 2.1 runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04 (this is what you wrote in your guide) - my config is the same of yours look better, and also with yours (tested in two persons) is not working - obviously it is modified in the epoch numbers, and I put the saving at each epoch, since I have a dataset of 900 images) and as regards the optimizer, I chose AdamW (but I repeat, while leaving AdaFactor with your parameters supplementary does not work). - I changed the sample prompt and the sampler for the prompt - I changed the learning rate (because with 900 images of the dataset) Everything else is identical to yours. But I repeat, even using yours, modifying ONLY the epochs (15) and saving at each epoch, it doesn't work. - I'm using Lora Tab

Samael1976

So there's a problem with RunPod... I'll do it all again and tell you. Because I assure you that I had taken your configuration file and modified by hand only the parts I needed. While the one I sent you is the one I "saved" and downloaded from the runpod server. I'll let you know soon. Thank you!

SECourses

whoever reads using fresh config from zip and loading into LoRA fixed issue for Samael. sometimes configs may get corrupted

Moonlly

Hi, how do you think higher training image will do? I tried your setting with 40 images and it works really well (100 epoc/4000steps). But now I want the model to have more expression of that person, aiming for 75-100 images. do you think the model needs to be about the same epoch? (100 epoc/7000 steps) just don't want it to be overfitted

Samael1976

yes totally! Thank you ❤️❤️❤️

Jan Zhor

i've gone step by step, but after clicking the "start training" button I got this, what should I do? To create a public link, set `share=True` in `launch()`. 15:00:19-460179 INFO Start training LoRA Flux1 ... 15:00:19-462150 INFO Validating lr scheduler arguments... 15:00:19-464150 INFO Validating optimizer arguments... 15:00:19-467724 INFO Validating C:/ace/kenta_joy_prepared\log existence and writability... SUCCESS 15:00:19-470153 INFO Validating C:/ace/kenta_joy_prepared\model existence and writability... SUCCESS 15:00:19-472159 INFO Validating C:/Kohya_GUI_Flux_v20/flux1-dev.safetensors existence... SUCCESS 15:00:19-474184 INFO Validating C:/ace/kenta_joy_prepared\img existence... SUCCESS 15:00:19-476155 INFO Folder 1_ohwx woman: 1 repeats found 15:00:19-478187 INFO Folder 1_ohwx woman: 55 images found 15:00:19-479183 INFO Folder 1_ohwx woman: 55 * 1 = 55 steps 15:00:19-480155 INFO Regulatization factor: 1 15:00:19-482156 INFO Total steps: 55 15:00:19-484155 INFO Train batch size: 1 15:00:19-485157 INFO Gradient accumulation steps: 1 15:00:19-487620 INFO Epoch: 100 15:00:19-488620 INFO max_train_steps (55 / 1 / 1 * 100 * 1) = 5500 15:00:19-489620 INFO stop_text_encoder_training = 0 15:00:19-490619 INFO lr_warmup_steps = 0 15:00:19-493620 INFO Saving training config to C:/ace/kenta_joy_prepared\model\kenta_flux1_ohwx_woman_v01_20240828-150019.json... 15:00:19-496620 INFO Executing command: c:\Kohya_GUI_Flux_v20\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no --dynamo_mode default --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 c:/Kohya_GUI_Flux_v20/kohya_ss/sd-scripts/flux_train_network.py --config_file C:/ace/kenta_joy_prepared\model/config_lora-20240828-150019.toml c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( Traceback (most recent call last): File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 70, in from tensorflow.python._pywrap_tensorflow_internal import * ImportError: DLL load failed while importing _pywrap_tensorflow_internal: A dynamic link library (DLL) initialization routine failed. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\transformers\utils\import_utils.py", line 1603, in _get_module return importlib.import_module("." + module_name, self.__name__) File "C:\Users\ace\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\transformers\models\clip\image_processing_clip.py", line 21, in from ...image_processing_utils import BaseImageProcessor, BatchFeature, get_size_dict File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\transformers\image_processing_utils.py", line 21, in from .image_transforms import center_crop, normalize, rescale File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\transformers\image_transforms.py", line 49, in import tensorflow as tf File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\tensorflow\__init__.py", line 38, in from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow # pylint: disable=unused-import File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 85, in raise ImportError( ImportError: Traceback (most recent call last): File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 70, in from tensorflow.python._pywrap_tensorflow_internal import * ImportError: DLL load failed while importing _pywrap_tensorflow_internal: A dynamic link library (DLL) initialization routine failed. Failed to load the native TensorFlow runtime. See https://www.tensorflow.org/install/errors for some common causes and solutions. If you need help, create an issue at https://github.com/tensorflow/tensorflow/issues and include the entire stack trace above this error message. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 710, in _get_module return importlib.import_module("." + module_name, self.__name__) File "C:\Users\ace\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion.py", line 20, in from transformers import CLIPImageProcessor, CLIPTextModel, CLIPTokenizer, CLIPVisionModelWithProjection File "", line 1075, in _handle_fromlist File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\transformers\utils\import_utils.py", line 1594, in __getattr__ value = getattr(module, name) File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\transformers\utils\import_utils.py", line 1593, in __getattr__ module = self._get_module(self._class_to_module[name]) File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\transformers\utils\import_utils.py", line 1605, in _get_module raise RuntimeError( RuntimeError: Failed to import transformers.models.clip.image_processing_clip because of the following error (look up to see its traceback): Traceback (most recent call last): File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 70, in from tensorflow.python._pywrap_tensorflow_internal import * ImportError: DLL load failed while importing _pywrap_tensorflow_internal: A dynamic link library (DLL) initialization routine failed. Failed to load the native TensorFlow runtime. See https://www.tensorflow.org/install/errors for some common causes and solutions. If you need help, create an issue at https://github.com/tensorflow/tensorflow/issues and include the entire stack trace above this error message. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "c:\Kohya_GUI_Flux_v20\kohya_ss\sd-scripts\flux_train_network.py", line 13, in from library import flux_models, flux_train_utils, flux_utils, sd3_train_utils, strategy_base, strategy_flux, train_util File "c:\Kohya_GUI_Flux_v20\kohya_ss\sd-scripts\library\flux_train_utils.py", line 17, in from library import flux_models, flux_utils, strategy_base, train_util File "c:\Kohya_GUI_Flux_v20\kohya_ss\sd-scripts\library\train_util.py", line 48, in from diffusers import ( File "", line 1075, in _handle_fromlist File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 701, in __getattr__ value = getattr(module, name) File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 701, in __getattr__ value = getattr(module, name) File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 700, in __getattr__ module = self._get_module(self._class_to_module[name]) File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 712, in _get_module raise RuntimeError( RuntimeError: Failed to import diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion because of the following error (look up to see its traceback): Failed to import transformers.models.clip.image_processing_clip because of the following error (look up to see its traceback): Traceback (most recent call last): File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 70, in from tensorflow.python._pywrap_tensorflow_internal import * ImportError: DLL load failed while importing _pywrap_tensorflow_internal: A dynamic link library (DLL) initialization routine failed. Failed to load the native TensorFlow runtime. See https://www.tensorflow.org/install/errors for some common causes and solutions. If you need help, create an issue at https://github.com/tensorflow/tensorflow/issues and include the entire stack trace above this error message. Traceback (most recent call last): File "C:\Users\ace\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\ace\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "c:\Kohya_GUI_Flux_v20\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['c:\\Kohya_GUI_Flux_v20\\kohya_ss\\venv\\Scripts\\python.exe', 'c:/Kohya_GUI_Flux_v20/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'C:/ace/kenta_joy_prepared\\model/config_lora-20240828-150019.toml']' returned non-zero exit status 1. 15:00:34-985849 INFO Training has ended.

SECourses

it is a library error not a particular error to fix. my suggestion to you would be uninstall your python and reinstall your python exactly as in this video, also install cuda and c++ tools and then reinstall kohya with newest zip file : https://www.youtube.com/watch?v=-NjNy7afOQ0&feature=youtu.be follow this tutorial video step by step

SECourses

i would say yes add more expressions and still go 100 epoch. it doesnt get overtrained quickly

Steve

Can't wait to try new configs, I'm always surprised you get amazing results without using any full length pictures. I'm surprised the model doesn't think you have no legs.

Hrabia M

Hi! i didn't have this error in previous version and configs.. what is the problem? 2024-08-29 00:05:48 INFO t5xxl_max_token_length: 512 flux_train_network.py:144 Traceback (most recent call last): File "/home/rafis88/Kohya/kohya_ss/sd-scripts/flux_train_network.py", line 445, in trainer.train(args) File "/home/rafis88/Kohya/kohya_ss/sd-scripts/train_network.py", line 258, in train tokenize_strategy = self.get_tokenize_strategy(args) File "/home/rafis88/Kohya/kohya_ss/sd-scripts/flux_train_network.py", line 145, in get_tokenize_strategy return strategy_flux.FluxTokenizeStrategy(t5xxl_max_token_length, args.tokenizer_cache_dir) File "/home/rafis88/Kohya/kohya_ss/sd-scripts/library/strategy_flux.py", line 27, in __init__ self.clip_l = self._load_tokenizer(CLIPTokenizer, CLIP_L_TOKENIZER_ID, tokenizer_cache_dir=tokenizer_cache_dir) File "/home/rafis88/Kohya/kohya_ss/sd-scripts/library/strategy_base.py", line 46, in _load_tokenizer tokenizer = model_class.from_pretrained(model_id, subfolder=subfolder) File "/home/rafis88/Kohya/kohya_ss/venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2255, in from_pretrained raise EnvironmentError( OSError: Can't load tokenizer for 'openai/clip-vit-large-patch14'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'openai/clip-vit-large-patch14' is the correct path to a directory containing all relevant files for a CLIPTokenizer tokenizer. Traceback (most recent call last): File "/home/rafis88/Kohya/kohya_ss/venv/bin/accelerate", line 8, in sys.exit(main()) File "/home/rafis88/Kohya/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main args.func(args) File "/home/rafis88/Kohya/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command simple_launcher(args) File "/home/rafis88/Kohya/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/home/rafis88/Kohya/kohya_ss/venv/bin/python', '/home/rafis88/Kohya/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', '/home/rafis88/Kohya/kohya_ss/dataset/images/img/model/config_lora-20240829-000541.toml']' returned non-zero exit status 1. 00:08:05-153682 INFO Training has ended.

visors

I tried the steps above but my kohya immediately crashes when trying to launch. Any ideas?

SECourses

it says it cant find your clip l file. give it accurately EnvironmentError( OSError: Can't load tokenizer for 'openai/clip-vit-large-patch14'.

Steven Nguyen

Can I use lora flux in forge sir?

Jan Zhor

do you have any flux comfyui basic workflow with maybe some prompt suggestion and lora option to start with? i was offline for past 3 weeks so I have to catch up

SECourses

yes you can but it uses more VRAM than swarmui and also i don't know if works with quantized models

esteban orozco

Hi! I'm running into some trouble with your script while training on my NVIDIA RTX 3080 (8GB VRAM). I'm using the Rank_9_7514MB.json config, which should work on my setup, but it keeps crashing. What's Happening I get a CUDA out-of-memory error when the script tries to load the T5-XXL text encoder onto the GPU, around 7.7 GB VRAM usage: text_encoders[1].to(accelerator.device, dtype=weight_dtype) What I've Tried So Far: Set lowvram to true, enabled gradient_checkpointing, and switched to FP8 precision for the text encoder. Turned on mem_eff_save and tried using split_mode and split_qkv to save memory. Reduced the network alpha and dimensions from 64 to 4 and 8, respectively. Also, tried enabling bucket_no_upscale and reducing max token length, but no luck. Any idea what else I can try to make it work on my 8GB VRAM? Thanks!

SECourses

are you using torch 2.5? if so can you make a new install and dont upgrade torch 2.5 and try again? also how much vram you are using before starting to train? did you check it? how much RAM you have? did you disable shared vram? i can connect your pc and check out few hours later please message me from discord

eduardo

Following your new tutorial video to install Kohyah, after i have selecte step one everything looks good except i do get this error when scrolling up ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 16.3 MB/s eta 0:00:00 ERROR: Could not find a version that satisfies the requirement xformers==0.0.27.post2 (from versions: none) ERROR: No matching distribution found for xformers==0.0.27.post2 the rest looks all good, should i be solving this issue before continuing with the installation?

SECourses

no it is fine. we don't use xformers for flux training. we use SDPA - torch optimization . it will eventually get fixed with next xformers version. it happens since we use latest torch :)

druhl

When downloading the Flux models using the windows download script, it says, "Could not set the permissions on the file 'C:\Kohya_GUI_Flux_Installer_21\.cache\huggingface\download\flux1-dev.safetensors.4610115bb0c89560703c892c59ac2742fa821e60ef5871b33493ba544683abd7.incomplete'. Error: [Errno 13] Permission denied: 'C:\\tmp_dc590ede-e5ad-4df7-b77a-40bb55a883b8'. Continuing without setting permissions." Is is okay to ignore this error?

Leonid Wolf

I have a problem installing Install kohya_ss GUI when I press 1 the command line crashes.

Leonid Wolf

I have a problem installing Install kohya_ss GUI when I press 1 the command line crashes.

SECourses

no it is not ok. please manually download from the links shared in the post. but permission error is bad you should fix it. either your windows account or your antivirus preventing

SECourses

please open a cmd and run the bat file there so we can see the reason. so open a cmd on the folder , type the name of the bat file fully and type enter

Albert

Will there be instructions on how to do the training on the Runpod?

SECourses

yes i have recorded the video for both massed compute and runpod. currently editing the video. hopefully will be available tomorrow on the channel

Albert

WOW! Thanks!

ProudChinaLover

I should have subscribed your patreon earlier. Very informative. Please keep this approach and make it not so technical and approachable. Thanks for your hard work. I love your explanations.

SECourses

with Rank_4_16960MB_Fast.json i get around 5-5.5 second / it with rtx 3090. use that config instead of rank 3. almost same quality. also ai-toolkit trains 512 px lower res pay attention to that. 512px reduces quality

SECourses

yes rank 3 and below uses fp8. fp16 requires min like 27 gb. also use base model file as fp16 it will be auto casted. i prefer that to be sure scripts working perfect

SECourses

thank you so much. runpod and massed compute tutorials already recorded hopefully will be published tomorrow

SECourses

you are welcome. sorry for delay hopefully tomorrow

SECourses

yes this is another reason good tips

howmuchsize

Hey, i got some problem with the lora i train following your method When i run it on comfyui the lora dont have any effect Do you have an idea why?

SECourses

can you try with swarmui? it is best one to test properly

ProudChinaLover

Please please please keep this dumb-proof approach. I love your approach: not too technical and explain stuff in detail. I really like it. I have 1 suggestion: you may have this inside your head while doing your recording; give a mini-answer/summary first before you explain a new aspect. (I think most of your parts are doing like this and I really like it). I always need some kinda non-technical explanations on how Generative AI works . Thanks for your hardwork. Please don't skip certain steps and assume everyone has watched everything of yours before . If you have some detailed explanations in another video, please show it on your youtube that I can refer to or put it inside the comment/description section. Thanks for your good work! I like it!

ProudChinaLover

I have 1 question, is there any Adetailer in Flux (as in Stable Diffusion) that can improve the 'face' of the subject?

Brett Baker

Here is the error: Kohya_ss setup menu: 1. Install kohya_ss GUI 2. (Optional) Install CuDNN files (to use the latest supported CuDNN version) 3. (DANGER) Install Triton 2.1.0 for Windows... only do it if you know you need it... might break training... 4. (Optional) Install specific version of bitsandbytes 5. (Optional) Manually configure Accelerate 6. (Optional) Launch Kohya_ss GUI in browser 7. Exit Setup Select an option: 1 09:02:47-811404 INFO Kohya_ss GUI version: v24.2.0 09:02:47-819404 INFO Python version is 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] 09:02:47-822404 ERROR The current version of python (sys.version_info(major=3, minor=9, micro=0, releaselevel='final', serial=0)) is not appropriate to run Kohya_ss GUI 09:02:47-824404 ERROR The python version needs to be greater or equal to 3.10.9 and less than 3.11.0 D:\Software\AI Software\Kohya_Flux\kohya_ss>

Brett Baker

It's an odd error since my python version is 3.10

Steve

Very nice! What are the configuration differences betwen rank 3-slow vs rank 4-fast? I can't find any config differences. Thanks!

RayHell

This is weird, I tried Rank_3_18246MB_Slow.json and the likeness is not good. I trained 300 epoch and tested every 20 epoch. It gets overfitted before I get a good likeness. On ai-toolkit I get an extremely good results with the exact same dataset.

SECourses

yes i already use it in swarm ui. it is photo of ohwx man

SECourses

the difference is T5 attention masking between fast and slow ones. it minorly improves quality but hugely reduces speed :)

SECourses

how many images you have and are you training 1024x1024? i did over 80 trainings and trained on another person - with 24 images worked perfect. so many of my users also got perfect results so far. can you message me from discord and give me details?

DGMDATAINTERACTIF

Hi ! Followed all the steps and I get this error after only 5 min of starting the training: D:\Kohya_GUI_Flux_Installer_21>Windows_Start_Kohya_SS.bat 20:44:45-377372 INFO headless: False 20:44:45-423277 INFO Using shell=True when running external commands... Running on local URL: http://127.0.0.1:7860 To create a public link, set `share=True` in `launch()`. 20:45:01-113093 INFO Loading config... 20:49:34-936952 INFO Removing existing directory D:/Kohya_GUI_Flux_Installer_21/train_imgs\img/1_ohwx man... 20:49:34-948916 INFO Copy D:/TRAINING 2 to D:/Kohya_GUI_Flux_Installer_21/train_imgs\img/1_ohwx man... 20:49:35-013720 INFO Regularization images directory is missing... not copying regularisation images... 20:49:35-015715 INFO Done creating kohya_ss training folder structure at D:/Kohya_GUI_Flux_Installer_21/train_imgs... 20:50:11-458173 INFO Start training LoRA Flux1 ... 20:50:11-459197 INFO Validating lr scheduler arguments... 20:50:11-461193 INFO Validating optimizer arguments... 20:50:11-463176 INFO Validating D:/Kohya_GUI_Flux_Installer_21/train_imgs\log existence and writability... SUCCESS 20:50:11-464157 INFO Validating D:/Kohya_GUI_Flux_Installer_21/train_imgs\model existence and writability... SUCCESS20:50:11-466179 INFO Validating D:/Kohya_GUI_Flux_Installer_21/flux1-dev.safetensors existence... SUCCESS 20:50:11-467176 INFO Validating D:/Kohya_GUI_Flux_Installer_21/train_imgs\img existence... SUCCESS 20:50:11-468173 INFO Folder 1_ohwx man: 1 repeats found 20:50:11-469170 INFO Folder 1_ohwx man: 15 images found 20:50:11-470168 INFO Folder 1_ohwx man: 15 * 1 = 15 steps 20:50:11-471166 INFO Regulatization factor: 1 20:50:11-473133 INFO Total steps: 15 20:50:11-474157 INFO Train batch size: 1 20:50:11-475127 INFO Gradient accumulation steps: 1 20:50:11-476125 INFO Epoch: 200 20:50:11-477122 INFO max_train_steps (15 / 1 / 1 * 200 * 1) = 3000 20:50:11-479116 INFO stop_text_encoder_training = 0 20:50:11-479116 INFO lr_warmup_steps = 0 20:50:11-481143 INFO Saving training config to D:/Kohya_GUI_Flux_Installer_21/train_imgs\model\Best_v2_20240901-205011.json... 20:50:11-483134 INFO Executing command: D:\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no --dynamo_mode default --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 D:/Kohya_GUI_Flux_Installer_21/kohya_ss/sd-scripts/flux_train_network.py --config_file D:/Kohya_GUI_Flux_Installer_21/train_imgs\model/config_lora-20240901-205011.toml D:\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( D:\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( D:\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( 2024-09-01 20:50:21 INFO Loading settings from D:/Kohya_GUI_Flux_Installer_21/train_imgs\model/config_lora-20240901-205011.toml... train_util.py:4189 INFO D:/Kohya_GUI_Flux_Installer_21/train_imgs\model/config_lora-20240901-205011 train_util.py:4208 highvram is enabled / highvramが有効です 2024-09-01 20:50:21 INFO t5xxl_max_token_length: 512 flux_train_network.py:144 D:\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884 warnings.warn( You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 INFO Using DreamBooth method. train_network.py:281 INFO prepare images. train_util.py:1803 INFO get image size from name of cache files train_util.py:1741 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 7424.42it/s] INFO set image size from cache files: 0/15 train_util.py:1748 INFO found directory D:\Kohya_GUI_Flux_Installer_21\train_imgs\img\1_ohwx man contains 15 image files train_util.py:1750 WARNING No caption file found for 15 images. Training will continue without captions for these images. If class token exists, it will be used. / train_util.py:1781 15枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャプションなしで学習を続行します。class tokenが存在する場合はそれを使います。 WARNING D:\Kohya_GUI_Flux_Installer_21\train_imgs\img\1_ohwx man\dan-1.png train_util.py:1788 WARNING D:\Kohya_GUI_Flux_Installer_21\train_imgs\img\1_ohwx man\dan-10.png train_util.py:1788 WARNING D:\Kohya_GUI_Flux_Installer_21\train_imgs\img\1_ohwx man\dan-11.png train_util.py:1788 WARNING D:\Kohya_GUI_Flux_Installer_21\train_imgs\img\1_ohwx man\dan-12.png train_util.py:1788 WARNING D:\Kohya_GUI_Flux_Installer_21\train_imgs\img\1_ohwx man\dan-13.png train_util.py:1788 WARNING D:\Kohya_GUI_Flux_Installer_21\train_imgs\img\1_ohwx man\dan-14.png... and 10 more train_util.py:1786 INFO 15 train images with repeating. train_util.py:1844 INFO 0 reg images. train_util.py:1847 WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:1852 INFO [Dataset 0] config_util.py:570 batch_size: 1 resolution: (1024, 1024) enable_bucket: False network_multiplier: 1.0 [Subset 0 of Dataset 0] image_dir: "D:\Kohya_GUI_Flux_Installer_21\train_imgs\img\1_ohwx man" image_count: 15 num_repeats: 1 shuffle_caption: False keep_tokens: 0 keep_tokens_separator: caption_separator: , secondary_separator: None enable_wildcard: False caption_dropout_rate: 0.0 caption_dropout_every_n_epoches: 0 caption_tag_dropout_rate: 0.0 caption_prefix: None caption_suffix: None color_aug: False flip_aug: False face_crop_aug_range: None random_crop: False token_warmup_min: 1, token_warmup_step: 0, alpha_mask: False, is_reg: False class_tokens: ohwx man caption_extension: .txt INFO [Dataset 0] config_util.py:576 INFO loading image sizes. train_util.py:876 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 326.75it/s] 2024-09-01 20:50:22 INFO prepare dataset train_util.py:884 INFO preparing accelerator train_network.py:335 accelerator device: cuda INFO Building Flux model dev flux_utils.py:45 INFO Loading state dict from D:/Kohya_GUI_Flux_Installer_21/flux1-dev.safetensors flux_utils.py:52 Traceback (most recent call last): File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "D:\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "D:\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "D:\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "D:\Kohya_GUI_Flux_Installer_21\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['D:\\Kohya_GUI_Flux_Installer_21\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/Kohya_GUI_Flux_Installer_21/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'D:/Kohya_GUI_Flux_Installer_21/train_imgs\\model/config_lora-20240901-205011.toml']' returned non-zero exit status 3221225477. 20:50:25-856086 INFO Training has ended. Any idea what I'm doing wrong ?

s h a r k e y

Currently testing Flux.Dev from a different angle, how fast can I train a likeness of a person with low quality images on a RTX 4090, with the goal of using as little images as possible and completing the training as quickly as I can. I'm happy with 512 resolution (for now) and I have managed to train at a batch size of 8 before it hitting shared ram and slowing to a crawl. I wonder how far flux can be pushed, 4 batch size with 0.001 for 50 epoch then retraining the best Lora output at 0.0001 for maybe 20 epoch has been pretty fast and the likeness is decent. I will test more and hopefully you can weigh in on what you might suggest *if we were to consider speed being priority over quality*

SECourses

this is totally a research topic. you need to test every case and compare. but after batch size 2 there weren't any actual speed improvement. have you tested it? bigger batch size = lower quality so if there isnt speed improvement you shouldn't increase batch size

SECourses

no difference. i just made it easier to do hugging face snapshot repo. i uploaded same files :D

s h a r k e y

I'll look into it first thing when I wake up 🤣 I'm sure you can agree this keeps you up all night. I did see you mention about not needing to go past batch size 2 so will need to look into it next. Ideally a lower batch size should give me greater accuracy on a smaller dataset (I think for styles or finetuning this might be worth while possibly) I will let you know my findings.

Vlad

Kohya_GUI_Flux_Installer_24 With "Rank_5_11700MB_12_40_Second_IT" have out of memory. I use GTX 1070 8GB for display and RTX 3080 12GB for training. 56GB RAM. torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 486.00 MiB. GPU 0 has a total capacity of 12.00 GiB of which 0 bytes is free. Of the allocated memory 9.76 GiB is allocated by PyTorch, and 1.05 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.

SECourses

11700 is very vram requiring may fail. are you monitoring the vram usage from nvitop? also extract config again and load into lora tab. make sure that never loaded into dreambooth. and enable back shared vram

Brett Baker

This is the error when I try to set up Kohya: Kohya_ss setup menu: 1. Install kohya_ss GUI 2. (Optional) Install CuDNN files (to use the latest supported CuDNN version) 3. (DANGER) Install Triton 2.1.0 for Windows... only do it if you know you need it... might break training... 4. (Optional) Install specific version of bitsandbytes 5. (Optional) Manually configure Accelerate 6. (Optional) Launch Kohya_ss GUI in browser 7. Exit Setup Select an option: 1 09:02:47-811404 INFO Kohya_ss GUI version: v24.2.0 09:02:47-819404 INFO Python version is 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] 09:02:47-822404 ERROR The current version of python (sys.version_info(major=3, minor=9, micro=0, releaselevel='final', serial=0)) is not appropriate to run Kohya_ss GUI 09:02:47-824404 ERROR The python version needs to be greater or equal to 3.10.9 and less than 3.11.0 D:\Software\AI Software\Kohya_Flux\kohya_ss>

Sabrina Usichenko

Hey, first of all, congratulations for this content. It's awesome. I've followed the guide and was able to configure and start the LORA's training process... however, the completion forecast when training a LORA with <10 face images is way far from my expectations: 24 hours to complete using RTX3060 8GB. I'm starting to guess if there's something wrong in my configuration step or if it's actually the expected time for 8GB VRAM GPU.

Sabrina Usichenko

Try with Python 3.10.11 (that's the one I'm using...) Tip: you can use PYENV (python version manager) to handle multiple python versions.

Michael

Hi, I really enjoy this tutorial, thank you. I have one question: In Kohya, under "Save every N epochs" I have it set to 20 with my total epoch being 100. Theoretically I should have 5 checkpoints, however at the end of training I only have one. No other checkpoints are being saved. Any ideas on what I might be getting wrong? I am running this with the new torch 2.5. Thank you

Robert Arsene

Hey! Can I ask why the trained Loras are over 2GB in size? Should it not be few hundred MB?

s h a r k e y

128 rank and dimensions, change to 64 for both for a decrease in size and some quality loss, I've even dropped to 32 for both and it's much smaller. 128 is best for quality 64 for a mixture of quality and size and 32 if you wanna prioritise size. Test all three (am I right in thinking you can resize the 128 to 64 later? This would avoid restraining)

SECourses

please refresh page and download v24. i think you checked while i was updating

SECourses

we save as float and 128 rank. if you mandatory need lower size, make save as fp16 - makes it 1 gb and also half rank makes it 512 mb

SECourses

true. addition to this save as fp16

SECourses

yes you need to have 5 checkpoints. can you send me your json file from discord so i can check

SECourses

are you using shared vram? for 8 gb to work you need to minimize VRAM usage. you need to have less than 500 mb usage before starting the training. can you verify both?

SECourses

i wonder if they properly test it. you can also do multi res training. generate multiple 1_ohwx man, set 1 folder 512x512, another one 768x768 and third one 1024x1024. it will train all. enable buckets too but lower resolution lowers the quality

Michael

Okay. I just added you on Discord. Once you accept I can send it to you. Unless you want me to just post it in a channel?

Michael

In case anyone was wondering, I had set my max train epochs to 35 when it should have been set to 0. Problem solved! Thanks Furkan!

Doc Snyder

I have had the same Error, the solution was: I used the "Dreambooth" Tab in Kohya no the "Lora" Tab .... :-D

jamzsam

For some reason before training starts it freezes on 'caching latents'. It started doing this since i installed v24. Have you any idea why? Thanks!

jamzsam

2024-09-03 04:47:20 INFO Using DreamBooth method. train_network.py:281 INFO prepare images. train_util.py:1803 INFO get image size from name of cache files train_util.py:1741 100%|████████████████████████████████████████████████████████████████████████████████| 23/23 [00:00<00:00, 3065.62it/s] INFO set image size from cache files: 0/23 train_util.py:1748 INFO found directory C:\SD\Training\KohySaraTrainingOutput\img6\1_sofia train_util.py:1750 vergara girl contains 23 image files INFO 23 train images with repeating. train_util.py:1844 INFO 0 reg images. train_util.py:1847 WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:1852 INFO [Dataset 0] config_util.py:570 batch_size: 1 resolution: (1024, 1024) enable_bucket: False network_multiplier: 1.0 [Subset 0 of Dataset 0] image_dir: "C:\SD\Training\KohySaraTrainingOutput\img6\1_sofia vergara girl" image_count: 23 num_repeats: 1 shuffle_caption: False keep_tokens: 0 keep_tokens_separator: caption_separator: , secondary_separator: None enable_wildcard: False caption_dropout_rate: 0.0 caption_dropout_every_n_epoches: 0 caption_tag_dropout_rate: 0.0 caption_prefix: None caption_suffix: None color_aug: False flip_aug: False face_crop_aug_range: None random_crop: False token_warmup_min: 1, token_warmup_step: 0, alpha_mask: False, is_reg: False class_tokens: sofia vergara girl caption_extension: .txt INFO [Dataset 0] config_util.py:576 INFO loading image sizes. train_util.py:876 100%|██████████████████████████████████████████████████████████████████████████████████████████| 23/23 [00:00 flux_utils.py:55 INFO prepare split model flux_train_network.py:99 INFO load state dict for lower flux_train_network.py:106 INFO load state dict for upper flux_train_network.py:111 INFO prepare upper model flux_train_network.py:114 2024-09-03 04:47:31 INFO split model prepared flux_train_network.py:129 INFO Building CLIP flux_utils.py:74 INFO Loading state dict from flux_utils.py:167 A:/Forge-flux/webui/models/text_encoder/clip_l.safetensors INFO Loaded CLIP: flux_utils.py:170 INFO Loading state dict from flux_utils.py:213 A:/Forge-flux/webui/models/text_encoder/t5xxl_fp16.safetensors INFO Loaded T5xxl: flux_utils.py:216 INFO Building AutoEncoder flux_utils.py:62 INFO Loading state dict from C:/Users/James/Documents/New folder/ae.safetensors flux_utils.py:66 INFO Loaded AE: flux_utils.py:69 import network module: networks.lora_flux INFO [Dataset 0] train_util.py:2326 INFO caching latents with caching strategy. train_util.py:984 INFO checking cache validity... train_util.py:994 100%|███████████████████████████████████████████████████████████████████████████████| 23/23 [00:00<00:00, 23001.67it/s] INFO caching latents... train_util.py:1038 0%| | 0/23 [00:00

EvilGiggles

I was training just fine with the previous versions, but the newest v24 version gets stuck at caching latents and never starts training. Even with the older configs.

楠 陈

Yes, the latest V24 version is also stuck and cannot be trained, W10 system. Hope this problem can be fixed

楠 陈

The V21 version can be trained normally, but subsequent versions cannot be trained normally.

SECourses

it auto handles. just make sure you dont use vram as much as possible and you have sufficient amount of virtual RAM set as shown in video

SECourses

this is due to accelerate please read section of the post : If Your Training Terminating at the Stage of Caching Latents

SECourses

if your error is same : this is due to accelerate please read section of the post : If Your Training Terminating at the Stage of Caching Latents

SECourses

this is due to accelerate please read section of the post : If Your Training Terminating at the Stage of Caching Latents

SECourses

this is due to accelerate please read section of the post : If Your Training Terminating at the Stage of Caching Latents

EvilGiggles

Edit: Actually it still doesn't work

SECourses

awesome. i saw someone also had fixed that way. so i say set restart computer and repeat until works :D

C

Hi, I've encountered this error during Step 1, how bad is it? How do I fix it? 13:32:06-340013 INFO Installing requirements from requirements_pytorch_windows.txt... Looking in indexes: https://download.pytorch.org/whl/cu124 Collecting torch==2.4.0+cu124 Using cached https://download.pytorch.org/whl/cu124/torch-2.4.0%2Bcu124-cp310-cp310-win_amd64.whl (2503.3 MB) Collecting torchvision==0.19.0+cu124 Using cached https://download.pytorch.org/whl/cu124/torchvision-0.19.0%2Bcu124-cp310-cp310-win_amd64.whl (5.9 MB) ERROR: Could not find a version that satisfies the requirement xformers==0.0.27.post2 (from versions: none) ERROR: No matching distribution found for xformers==0.0.27.post2

SECourses

second part Update_Kohya_and_Fix_FLUX_Step2.bat fixes this issue

C

Thank you!

SECourses

you are welcome also download v25 configs please

SECourses

please download v25 configs it should fix

SECourses

please download v25 configs it should fix

SECourses

please download v25 configs it should fix

SECourses

please download v25 configs it should fix

SECourses

please download v25 configs it should fix

SECourses

thank you so much i will try to follow your suggestions

Kornel Hartung

I was able to get it working under Linux, as well. Torch 2.4.0+cu124 works fine, Python version is 3.10.14 (main, Aug 5 2024, 02:53:45) [GCC 13.2.0] I have went thorough your bat files, and executed them in the same order. Training speed is 20.5s/it on a secondary (no graphics) 3060 12GB using rank_5 (11489MiB / 12288MiB)

Manpreet Singh

Looks like the configs have max bucket resolution set to 2048 and you also crop+resize your images to 1024x1024. Does Kohya ever use the 2048 bucket in this case? Or this effectively using 1024 as max bucket resolution?

jamzsam

Thanks. It now works.

SECourses

it doesnt use since bucketing not enabled. even if you enable it only downscale unless you enable upscale option which i dont know how :D

SECourses

by the way just fixed rank4 so download newest zip file

Manpreet Singh

So you suggest training at 1024x1024 only is better than bucketing across resolutions?

SECourses

good question but i didnt test impact of it since i don't have a good such dataset.

SECourses

rank 4 was broken previously and yes currently fixed. rank 3 has it on but it slows down hugely. i think it is also bug and i am expecting a fix soon hopefully

Tiago4D

Hi Dr.! How I can do my training with flux1-dev-bnb-nf4 model? is possible?

SECourses

Only FP8 and FP16 base models supported. Use FP16 base model it gets auto cast

chucky inaba

Hey, ive been using v16 of the khoya trainer and thats been working fine but everytime i try to use the any of the later version it refuses to run off the gpu and instead uses my cpu. i checked the params in khoya but im still not sure whats causing it. help would be much appreciated, thanks.

JackOppss

Hi and thank u for ur effort, I followed your tutorial and tried one of the lowest conf still not working. I have an RTX 4090. here what I am getting: 09:58:00-132843 INFO headless: False 09:58:00-172582 INFO Using shell=True when running external commands... Running on local URL: http://127.0.0.1:7860 To create a public link, set `share=True` in `launch()`. 09:58:14-090516 INFO Loading config... 09:59:11-912756 INFO Copy D:/NJK to D:/Kohya_GUI_Flux_Installer_25/OUTPUT/01\img/40_NJK man... 09:59:11-966755 INFO Regularization images directory is missing... not copying regularisation images... 09:59:11-968755 INFO Done creating kohya_ss training folder structure at D:/Kohya_GUI_Flux_Installer_25/OUTPUT/01... 09:59:48-585619 INFO Start training Dreambooth... 09:59:48-586619 INFO Validating lr scheduler arguments... 09:59:48-587619 INFO Validating optimizer arguments... 09:59:48-587619 INFO Validating D:/Kohya_GUI_Flux_Installer_25/OUTPUT/01\log existence and writability... SUCCESS 09:59:48-588618 INFO Validating D:/Kohya_GUI_Flux_Installer_25/OUTPUT/01\model existence and writability... SUCCESS 09:59:48-589620 INFO Validating D:/Kohya_GUI_Flux_Installer_25/flux1-dev.safetensors existence... SUCCESS 09:59:48-590619 INFO Validating D:/Kohya_GUI_Flux_Installer_25/OUTPUT/01\img existence... SUCCESS 09:59:48-590619 INFO Folder 40_NJK man: 40 repeats found 09:59:48-591619 INFO Folder 40_NJK man: 14 images found 09:59:48-592619 INFO Folder 40_NJK man: 14 * 40 = 560 steps 09:59:48-592619 INFO Regulatization factor: 1 09:59:48-593619 INFO Total steps: 560 09:59:48-593619 INFO Train batch size: 1 09:59:48-594619 INFO Gradient accumulation steps: 1 09:59:48-595619 INFO Epoch: 200 09:59:48-595619 INFO max_train_steps (560 / 1 / 1 * 200 * 1) = 112000 09:59:48-596618 INFO lr_warmup_steps = 0 09:59:48-597618 INFO Saving training config to D:/Kohya_GUI_Flux_Installer_25/OUTPUT/01\model\Rank_10_20240904-095948.json... 09:59:48-598619 INFO Executing command: D:\Kohya_GUI_Flux_Installer_25\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no --dynamo_mode default --gpu_ids 0 --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 D:/Kohya_GUI_Flux_Installer_25/kohya_ss/sd-scripts/flux_train.py --config_file D:/Kohya_GUI_Flux_Installer_25/OUTPUT/01\model/config_dreambooth-20240904-095948.toml D:\Kohya_GUI_Flux_Installer_25\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( D:\Kohya_GUI_Flux_Installer_25\kohya_ss\venv\lib\site-packages\xformers\ops\fmha\flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch. @torch.library.impl_abstract("xformers_flash::flash_fwd") D:\Kohya_GUI_Flux_Installer_25\kohya_ss\venv\lib\site-packages\xformers\ops\fmha\flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch. @torch.library.impl_abstract("xformers_flash::flash_bwd") D:\Kohya_GUI_Flux_Installer_25\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( 2024-09-04 09:59:55 INFO Loading settings from train_util.py:4189 D:/Kohya_GUI_Flux_Installer_25/OUTPUT/01\model/config_dreambooth-2024090 4-095948.toml... INFO D:/Kohya_GUI_Flux_Installer_25/OUTPUT/01\model/config_dreambooth-2024090 train_util.py:4208 4-095948 2024-09-04 09:59:55 INFO Using DreamBooth method. flux_train.py:101 INFO prepare images. train_util.py:1803 INFO get image size from name of cache files train_util.py:1741 100%|███████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<00:00, 13964.39it/s] INFO set image size from cache files: 0/14 train_util.py:1748 INFO found directory D:\Kohya_GUI_Flux_Installer_25\OUTPUT\01\img\40_NJK man train_util.py:1750 contains 14 image files WARNING No caption file found for 14 images. Training will continue without train_util.py:1781 captions for these images. If class token exists, it will be used. / 14枚の画像にキャプションファイルが見つかりませんでした。これらの画像につ いてはキャプションなしで学習を続行します。class tokenが存在する場合はそれを使います。 WARNING D:\Kohya_GUI_Flux_Installer_25\OUTPUT\01\img\40_NJK man\NJ (1).jpg train_util.py:1788 WARNING D:\Kohya_GUI_Flux_Installer_25\OUTPUT\01\img\40_NJK man\NJ (10).jpg train_util.py:1788 WARNING D:\Kohya_GUI_Flux_Installer_25\OUTPUT\01\img\40_NJK man\NJ (11).jpg train_util.py:1788 WARNING D:\Kohya_GUI_Flux_Installer_25\OUTPUT\01\img\40_NJK man\NJ (12).jpg train_util.py:1788 WARNING D:\Kohya_GUI_Flux_Installer_25\OUTPUT\01\img\40_NJK man\NJ (13).jpg train_util.py:1788 WARNING D:\Kohya_GUI_Flux_Installer_25\OUTPUT\01\img\40_NJK man\NJ (14).jpg... train_util.py:1786 and 9 more INFO 560 train images with repeating. train_util.py:1844 INFO 0 reg images. train_util.py:1847 WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:1852 INFO [Dataset 0] config_util.py:570 batch_size: 1 resolution: (896, 896) enable_bucket: False network_multiplier: 1.0 [Subset 0 of Dataset 0] image_dir: "D:\Kohya_GUI_Flux_Installer_25\OUTPUT\01\img\40_NJK man" image_count: 14 num_repeats: 40 shuffle_caption: False keep_tokens: 0 keep_tokens_separator: caption_separator: , secondary_separator: None enable_wildcard: False caption_dropout_rate: 0 caption_dropout_every_n_epoches: 0 caption_tag_dropout_rate: 0.0 caption_prefix: None caption_suffix: None color_aug: False flip_aug: False face_crop_aug_range: None random_crop: False token_warmup_min: 1, token_warmup_step: 0, alpha_mask: False, is_reg: False class_tokens: NJK man caption_extension: .txt INFO [Dataset 0] config_util.py:576 INFO loading image sizes. train_util.py:876 100%|█████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<00:00, 466.67it/s] INFO prepare dataset train_util.py:884 INFO prepare accelerator flux_train.py:171 accelerator device: cuda INFO Building AutoEncoder flux_utils.py:62 INFO Loading state dict from D:/Kohya_GUI_Flux_Installer_25/ae.safetensors flux_utils.py:66 INFO Loaded AE: flux_utils.py:69 2024-09-04 09:59:56 INFO [Dataset 0] train_util.py:2326 INFO caching latents with caching strategy. train_util.py:984 INFO checking cache validity... train_util.py:994 100%|██████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 2024-09-04 09:59:59 INFO Building CLIP flux_utils.py:74 INFO Loading state dict from D:/Kohya_GUI_Flux_Installer_25/clip_l.safetensors flux_utils.py:167 INFO Loaded CLIP: flux_utils.py:170 INFO Loading state dict from flux_utils.py:213 D:/Kohya_GUI_Flux_Installer_25/t5xxl_fp16.safetensors 2024-09-04 10:00:00 INFO Loaded T5xxl: flux_utils.py:216 2024-09-04 10:00:06 INFO [Dataset 0] train_util.py:2347 INFO caching Text Encoder outputs with caching strategy. train_util.py:1107 INFO checking cache validity... train_util.py:1113 100%|██████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00 flux_utils.py:55 FLUX: Gradient checkpointing enabled. CPU offload: False INFO enable block swap: double_blocks_to_swap=0, single_blocks_to_swap=0 flux_train.py:272 number of trainable parameters: 11901408320 prepare optimizer, data loader etc. INFO use Adafactor optimizer | {'scale_parameter': False, 'relative_step': train_util.py:4500 False, 'warmup_init': False, 'weight_decay': 0.01} WARNING because max_grad_norm is set, clip_grad_norm is enabled. consider set to train_util.py:4528 0 / max_grad_normが設定されているためclip_grad_normが有効になります。0に設定 して無効にしたほうがいいかもしれません WARNING constant_with_warmup will be good / train_util.py:4532 スケジューラはconstant_with_warmupが良いかもしれません enable full bf16 training. running training / 学習開始 num examples / サンプル数: 560 num batches per epoch / 1epochのバッチ数: 560 num epochs / epoch数: 200 batch size per device / バッチサイズ: 1 gradient accumulation steps / 勾配を合計するステップ数 = 1 total optimization steps / 学習ステップ数: 112000 steps: 0%| | 0/112000 [00:00

Mikkel Olsen

How do you update from one version of the zip to a newer one? Can you overwrite the files in the folder or do you need to start all over with install when unzipping?

SECourses

yes overwrite existing files. also before running kohya use Update_Kohya_and_Fix_FLUX_Step2.bat to get latest version

SECourses

hi you have 112k steps. also i think you have at least once loaded the config into the dreambooth tab and corrupted. so you have lots of error first get a fresh config from zip file second make your folder repeat 1 and it should work perfect on rtx 4090. you can use rank 3 or rank 4

C

Hi, I followed your tutorial, whenevr I try to train anything, I get to this, essentially it gets to "caching latents" and then nothing happens and says "training has ended." What am I doing wrong? Also initially when I load the GUY, I get this which says GPU not available, might that be it? A start up: 15:38:05-868768 INFO Kohya_ss GUI version: v24.2.0 15:38:06-183803 INFO Submodule initialized and updated. 15:38:06-185802 INFO nVidia toolkit detected 15:38:07-127402 INFO Torch 2.4.0+cu124 15:38:07-148194 WARNING Torch reports GPU not available Here is when trying to train: INFO [Dataset 0] config_util.py:576 INFO loading image sizes. train_util.py:876 100%|████████████████████████████████████████████████████████████████████████████████| 59/59 [00:00<00:00, 58794.00it/s] INFO prepare dataset train_util.py:884 INFO network for CLIP-L only will be trained. T5XXL will not be trained flux_train_network.py:50 / CLIP-Lのネットワークのみが学習されます。T5XXLは学習されません INFO preparing accelerator train_network.py:335 accelerator device: cpu INFO Building Flux model dev flux_utils.py:45 INFO Loading state dict from flux_utils.py:52 C:/Kohya_GUI_Flux_Installer_24/flux1-dev.safetensors INFO Loaded Flux: flux_utils.py:55 INFO Building CLIP flux_utils.py:74 INFO Loading state dict from C:/Kohya_GUI_Flux_Installer_24/clip_l.safetensors flux_utils.py:167 INFO Loaded CLIP: flux_utils.py:170 INFO Loading state dict from flux_utils.py:213 C:/Kohya_GUI_Flux_Installer_24/t5xxl_fp16.safetensors INFO Loaded T5xxl: flux_utils.py:216 INFO Building AutoEncoder flux_utils.py:62 INFO Loading state dict from C:/Kohya_GUI_Flux_Installer_24/vae.safetensors flux_utils.py:66 INFO Loaded AE: flux_utils.py:69 import network module: networks.lora_flux INFO [Dataset 0] train_util.py:2326 INFO caching latents with caching strategy. train_util.py:984 INFO checking cache validity... train_util.py:994 100%|███████████████████████████████████████████████████████████████████████████████████████████| 59/59 [00:00

Tiago4D

Sorry, where to change?

SECourses

on the gui look fp8 : fp8 base unet

Tiago4D

2024-09-04 10:39:58 INFO prepare split model flux_train_network.py:99 INFO load state dict for lower flux_train_network.py:106 INFO load state dict for upper flux_train_network.py:111 INFO prepare upper model flux_train_network.py:114 Traceback (most recent call last): File "C:\kohya_ss\kohya_ss\sd-scripts\flux_train_network.py", line 446, in trainer.train(args) File "C:\kohya_ss\kohya_ss\sd-scripts\train_network.py", line 344, in train model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator) File "C:\kohya_ss\kohya_ss\sd-scripts\flux_train_network.py", line 83, in load_target_model model = self.prepare_split_model(model, weight_dtype, accelerator) File "C:\kohya_ss\kohya_ss\sd-scripts\flux_train_network.py", line 116, in prepare_split_model flux_upper.to(accelerator.device, dtype=target_dtype) File "C:\kohya_ss\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1174, in to return self._apply(convert) File "C:\kohya_ss\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 780, in _apply module._apply(fn) File "C:\kohya_ss\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 805, in _apply param_applied = fn(param) File "C:\kohya_ss\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1167, in convert raise NotImplementedError( NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device. Traceback (most recent call last): File "C:\Users\tiago\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\tiago\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\kohya_ss\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "C:\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "C:\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "C:\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\\kohya_ss\\kohya_ss\\venv\\Scripts\\python.exe', 'C:/kohya_ss/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'F:/OneDrive/_00_Tiago4D/Pol/Jobs/Titi/lora\\model/config_lora-20240904-103947.toml']' returned non-zero exit status 1. 10:39:59-899928 INFO Training has ended.

Brett Kelly

Training went okay but when I try and do a prompt and apply the Lora the model does not seem to use the lora, any ideas what may be causing this?

Tiago4D

is not working, need to change on step 01?

inès Marzat

Hey! Thanks a lot for doing all those amazing tutorials. I've successfully installed Kohya, trained my Lora Models, installed SwarmUI, Flux models appear in the interface, my Loras too, everything seems to be in the right place and folder BUT my lora models doesn't seem to affect the generated image (°_°). I've tried many different things but I don't know what to do anymore. Could you help me figure this out ? Many thanks!

inès Marzat

I think I have the same issue :(

SECourses

well without seeing your entire training workflow it is impossible to know :/ i did over 90 trainings all worked :/ what did you train a person object style?

SECourses

well without seeing your entire training workflow it is impossible to know :/ i did over 90 trainings all worked :/ what did you train a person object style?

SECourses

what is your GPU? your problem is "says GPU not available"

inès Marzat

I guessed :'( I trained a model with my own images. I'm an artist and I do 3D abstract faces kind of... So it's kind of persons, kind of concepts. When I followed you tutorials for training with stable diffusion on ComfyUI I didn't have that issue tho. Should I try to test my Loras on ComfyUI maybe ? I tried it quickly but it didn't recognise the flux 1dev model. Do you have any tutorials on that ? big thanks for your help!

inès Marzat

Just ran Comfy UI after having updated it and it seems it doesn't apply my other Loras either anymore :( very weird, some weeks/months ago it worked perfectly! Would the problem come from the update ? something external ? Any clue would help a lot.

Paco Chanivet

Hello, I follow the tutorial step by step but I get this error message when installing Kohya_ss: INFO Kohya_ss GUI version: v24.2.0 INFO Python version is 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit4)] INFO Submodule initialized and updated. INFO Installing requirements from requirements_pytorch_windows.txt... Looking in indexes: https://download.pytorch.org/whl/cu124 Collecting torch==2.4.0+cu124 Using cached https://download.pytorch.org/whl/cu124/torch-2.4.0%2Bcu124-cp310-cp310-win_amd64.whl (2503.3 MB) Collecting torchvision==0.19.0+cu124 Using cached https://download.pytorch.org/whl/cu124/torchvision-0.19.0%2Bcu124-cp310-cp310-win_amd64.whl (5.9 MB) ERROR: Could not find a version that satisfies the requirement xformers==0.0.27.post2 (from versions: none) ERROR: No matching distribution found for xformers==0.0.27.post2 What do I have to do?

SECourses

hello just proceed with step 2 installer it fixes it

Hans Peter

Training does not start. It loads something into the normal ram but does not transfer it to the graphics card. INFO Loaded T5xxl: flux_utils.py:216 INFO Building AutoEncoder flux_utils.py:62 INFO Loading state dict from flux_utils.py:66 C:/Users/xxx/pinokio/api/comfyui.git/app/models/vae/ae.safetensors INFO Loaded AE: flux_utils.py:69 import network module: networks.lora_flux INFO [Dataset 0] train_util.py:2326 INFO caching latents with caching strategy. train_util.py:984 INFO checking cache validity... train_util.py:994 100%|██████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00

Hans Peter

And the parameters are also not taken over by Kohya from the config file is always set to None

Tiago4D

on this case its working, but my machine gpu is 3060... I´m trying to reduce the time of calculations, is possible?

SECourses

You are using pinokio install as in tutorials

SECourses

Don't use pinokio follow tutorials

SECourses

Not very much since not fitting into vram. You can reduce resolution to 512px for faster but it will reduce quality. Or use cloud services just published a full tutorial for it

Tiago4D

Thanks Dr.!

SECourses

You are welcome thanks for support

inès Marzat

I've just made an experiment to see if the problem would come from my trained Lora files, downloaded a Lora Flux model on civitai, added it in SwarmUI, but same, it didn't apply the Lora on my generated images :( The Lora file is selected, Flux model as well, followed your tut for parameters... Do you have any ideas of where the problem could come from ? Things I should test or try to fix it ? Especially now that it's also weirdly affecting my Comfy UI installation that used to work well before I updated it ! Thanks again for your help.

SECourses

Sadly hard to know without seeing entire process :/ if you upgrade gold level I can connect your pc and check

Hans Peter

Doesn't change anything if I load the models from a different folder. The problem remains.

SECourses

Sent a private message please check it out. If you still can't make upgrade gold tier I will connect pc and setup for you

Hans Peter

That's nice but you don't need it. I've got it running on my other system. It just doesn't run on my main computer.

SECourses

That is great so something causing. Sadly I don't know atm from this info

Brett Kelly

person training, I cant see anything in the logs for swarmui about using the lora. I have an 8GB card so maybe some issue with this...

HirumaDiabe

Thank you so much, it worked for me with Massed Compute. The only thing that didn't work was when I tried to upload my Flux Lora model to Hugging Face using JupyterLab to download it to my computer. So what I do is connect to an online cloud service (like pCloud, OneDrive, etc.) and upload my Loras there to download them later to my computer. Apart from that, everything was good. Thanks again !

SECourses

yes you can use any cloud service. great

Brett Kelly

I think we need to use GGUF model for loading Lora with low vram cards, maybe you can do tutorial about this?

chucky inaba

I've tried it with v27 and i've set the accelerate, ive reset it to be sure and the problem still exists but v16 still works. thank you for your response

SECourses

well if you upgrade gold member i can connect your pc and check. nothing else comes to my mind

楠 陈

I think it is the training after adding CLIP_l that makes LORA invalid. I also encountered the problem that the LORA model is invalid after loading the LORA file.

SECourses

this is a valid point. it works in swarmui but i didnt test others yet

Brett Kelly

Pretty sure we need to use GGUF model on 8GB, I can't work out how to use it on swarmUI though but reading from others need to use that for lowvram

SECourses

For inference yes that is better. But I haven't tested loras with them yet. So don't know if working

inès Marzat

Just updated my membership to gold so you can do that ^^ How should we proceed so you can check my PC?

SECourses

awesome please message me from discord and install google remote

Albert

Hi! I have a bit of skill success training on sdxl, but now I trained on fluxdev on runpod as per instructions and after training added lora to comfyui and at gyneration error ora key not loaded: lora_te1_text_model_encoder_layers_9_self_attn_v_proj.alpha lora key not loaded: lora_te1_text_model_encoder_layers_9_self_attn_v_proj.lora_down.weight lora key not loaded: lora_te1_text_model_encoder_layers_9_self_attn_v_proj.lora_up.weight

Albert

I trained on runpod and on windows, same error.

Albert

I downloaded forge, checked it with different models, the same thing is just being generated and lora is not taken into account, it does not work.

SECourses

message me from discord and show all settings screenshots. i did over 100 trainings not a once that error

100Sachen80

OneTrainer now also supports Flux Dev Model. Did you try that? Are you planing to release a tutorial on that for different GPU (mine is 3060 12GB), or would it be better just to stick with Khoya tutorial? Thanks for all your great work!

SECourses

yes i plan to cover onetrainer too. you can actually copy paste our config there i mean look each variable but i didnt have chance yet

AiAiAE_Studio

HI! Cant load config into gui. No browser windows for selecting config files. On the right top is error: error Connection errored out. Please help What I've tried: 1) reunstall 2) different browser 3) update pydantic logs: File "c:\ai\onetrainer\venv\lib\site-packages\pydantic\_internal\_generate_schema.py", line 789, in _generate_schema_inner return self.match_type(obj) File "c:\ai\onetrainer\venv\lib\site-packages\pydantic\_internal\_generate_schema.py", line 875, in match_type return self._unknown_type_schema(obj) File "c:\ai\onetrainer\venv\lib\site-packages\pydantic\_internal\_generate_schema.py", line 415, in _unknown_type_schema raise PydanticSchemaGenerationError( pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for . Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it. If you got this error by calling handler() within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema()` since we do not call `__get_pydantic_core_schema__` on `` otherwise to avoid infinite recursion. For further information visit https://errors.pydantic.dev/2.8/u/schema-for-unknown-type

Sonivas Sx

Thanks for everything! Doing my first training on a 4080 i used Rank 5, can push it to rank 4 ? Also should i download CuDNN? For a dataset of 24 images it takes around 14hrs. Any suggestions you can give me or information i can give you to make this faster i'd be glad to help

Cemil Hacimahmutoglu

I got the same error. I reinstalled 2 times and the result is the same

SECourses

i am about to make a new update. how much VRAM you had? if 16 gb it can benefit you a lot

SECourses

activate venv and do pip install fastapi==0.111.0 . i will tell kohya to make fix asap . it is error of fastapi

SECourses

i just updated v28 and step 2 will fix it. it is fastapi error

SECourses

i just updated v28 and step 2 will fix it. it is fastapi error

Michael

Thank you so much for doing all this work. Amazing work I love learning all of these new technologies but just don't have the time between family and work to dedicate as much time as I would like. So thank you for helping me keep up 😌🙏 Any chance you could try providing a couple of different datasets with example captioning? I know you are using the default dataset for consistent testing I would just like to see other datasets too because sometimes I can't tell if my datasets is a problem or my captioning :)

Sonivas Sx

Great! my VRAM is 16GB. Do you mean to move to rank 4 or download CuDNN or both ?

SECourses

well i tested configs and sadly didnt reduce that much. check the new configs. CuDNN doesnt make difference

Raffaele Massaro

first of all thanks for these tutorials, I followed everything perfectly, but with a geforce 3060 with 12gb and 32gb of ram with rank 5 it doesn't go below 26s/it and with rank 6 24s/it ... where am I wrong?

SECourses

you are not wrong. with torch 2.5 , empty second rtx 3060 i get 20 second / it best. that is the speed loss due to having low VRAM

guangyu niu

Hi, I'm trying to train a flux lora base on a flux finetune model, however, when I use koyha script with this finetune model, it report a error I never seen before which is "NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device." any idea why this happen? model link: https://www.liblib.art/modelinfo/ad11ceec4e26466b8da847bd8f6b4068?from=search

SECourses

use my model downloader. use downloaded models

Ichi Shun

If I want to train a hoodie with a printed image, should I use a product photo or a product photo with the model wearing it? Is there any advice for this situation?

SECourses

include both but dont have same person images multiple time. so it will only learn hoodie not the person

Ichi Shun

With 1 product, how many different people will be fine? In case the hoodie has printing on both the front and back, when training, I should train it twice for the front and back or put it together in 1 training.

SECourses

there is not a precise number but for flux more better i see that

guangyu niu

just wanna figure out why this model not working

Albert

Thanks, I set the settings correctly, the training goes on the base model Flux dev, when I tried to train on other models, the training does not start immediately error, I tried 3 different models and does not train, trains only on the base model Flux dev? What is the reason? Have you tried training on other models?

SECourses

i trained fp16 dev and fp16 schnell model not others . if you upgrade to gold i can connect your pc and determine your error.

Albert

Thanks, I'll update next month. I mean I downloaded other flux models on civitai.com, with these flux models the Lora training doesn't run, but with the basic flux dev runs. You said in the video that you can train on other models, I checked and practicing on other models is not possible. This is not only my problem, but also future people who will train on your parameters will not be able to train on other models. I checked three different models, it seemed to me either you have a bug or a defense.

SECourses

are other models on civitai original fp8 or fp16? they are probably not. they need to be like either 23.8 gb or 11.8 gb

Albert

All of my downloaded 20-11 GB DEV models. For example STOIQO NewReality Flux Dev.

SECourses

well its size is not same as dev model. so something is off. that is why kohya probably failing. " Clips, t5 and VAE are included in the models." you can open an issue thread on kohya and i am sure he can fix

Albert

Ok, thanks.

SECourses

you are welcome. i think support for these models will come eventually

Ichi Shun

Hello, I run on runpod, I get this error. How can I fix this? INFO [Dataset 0] config_util.py:576 INFO loading image sizes. train_util.py:876 0it [00:00, ?it/s] INFO make buckets train_util.py:882 WARNING min_bucket_reso and max_bucket_reso are ignored if train_util.py:899 bucket_no_upscale is set, because bucket reso is defined by image size automatically / bucket_no_upscaleが指定された場合は、bucketの解像度は画 像サイズから自動計算されるため、min_bucket_resoとmax_bu cket_resoは無視されます INFO number of images (including repeats) / train_util.py:928 各bucketの画像枚数(繰り返し回数を含む) /workspace/kohya_ss/venv/lib/python3.10/site-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice. return _methods._mean(a, axis=axis, dtype=dtype, /workspace/kohya_ss/venv/lib/python3.10/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide ret = ret.dtype.type(ret / rcount) INFO mean ar error (without repeats): nan train_util.py:938 ERROR No data found. Please verify arguments train_network.py:332 (train_data_dir must be the parent of folders with images) / 画像がありません。引数指定を確認してください(train_ data_dirには画像があるフォルダではなく、画像があるフ ォルダの親フォルダを指定する必要があります)

SECourses

incorrect folder setup. please watch the relevant part of the tutorial and reprepare your training dataset folders

Michael

I think I broke something -.- I made a great model and then I updated to v30 and I thought I updated all other files but am now getting this message when I try to start training: Traceback (most recent call last): File "C:\Kohya\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\flux_train_network.py", line 519, in trainer.train(args) File "C:\Kohya\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\train_network.py", line 441, in train self.post_process_network(args, accelerator, network, text_encoders, unet) File "C:\Kohya\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\flux_train_network.py", line 170, in post_process_network self.train_t5xxl = network.train_t5xxl File "C:\Kohya\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1729, in __getattr__ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'") AttributeError: 'LoRANetwork' object has no attribute 'train_t5xxl' Traceback (most recent call last): File "C:\Users\Aprucia\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\Aprucia\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Kohya\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "C:\Kohya\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "C:\Kohya\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "C:\Kohya\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\\Kohya\\Kohya_GUI_Flux_Installer_30\\kohya_ss\\venv\\Scripts\\python.exe', 'C:/Kohya/Kohya_GUI_Flux_Installer_30/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'C:/Users/Aprucia/Desktop/AIModel training/Image/Model_Batches/Ted_Wheeler/Training_destination\\model/config_lora-20240907-142501.toml']' returned non-zero exit status 1.

SECourses

hello i just tested our configs on the newest version. loaded into lora tab directly and set my paths. it works perfect no issues. so my guess is that you loaded the config at least 1 time into dreambooth tab and corrupted it. get a fresh config load into lora tab and re-set - also dont enable Train T5-XXL

ProudChinaLover

For Mass Compute, is it being powered by the most updated Torch so that I can train much faster?

SECourses

use the Massed_Compute_Kohya_FLUX_Instructions.txt file and it will upgrade kohya and torch to 2.4 and yes it works max performance

shigeto

CUDA Out of Memory during Caching Latents Stage - Assistance Needed Hello, Here’s what I’ve tried so far: Verified accelerate settings Followed the tutorial: https://youtu.be/adVhm9aI9Gc Replaced my YAML file with the recommended one in the Hugging Face accelerate folder I am encountering a persistent issue during training at the stage of caching latents. Despite having sufficient GPU memory (23.99 GiB total, 19.18 GiB free), the process fails with a CUDA Out of Memory error. Below is a snippet of the error message: INFO caching latents... train_util.py:1038 0%| | 0/5 [00:00 trainer.train(args) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\train_network.py", line 387, in train train_dataset_group.new_cache_latents(vae, accelerator.is_main_process) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\library\train_util.py", line 2325, in new_cache_latents dataset.new_cache_latents(model, is_main_process) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\library\train_util.py", line 1041, in new_cache_latents caching_strategy.cache_batch_latents(model, batch, subset.flip_aug, subset.alpha_mask, subset.random_crop) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\library\strategy_flux.py", line 230, in cache_batch_latents self._default_cache_batch_latents( File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\library\strategy_base.py", line 279, in _default_cache_batch_latents latents_tensors = encode_by_vae(img_tensor).to("cpu") File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\library\strategy_flux.py", line 226, in encode_by_vae = lambda img_tensor: vae.encode(img_tensor).to("cpu") File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\library\flux_models.py", line 343, in encode z = self.reg(self.encoder(x)) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\library\flux_models.py", line 196, in forward h = self.down[i_level].block[i_block](hs[-1]) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\library\flux_models.py", line 104, in forward h = swish(h) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\sd-scripts\library\flux_models.py", line 54, in swish return x * torch.sigmoid(x) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 19.18 GiB is free. Of the allocated memory 3.18 GiB is allocated by PyTorch, and 26.62 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) Traceback (most recent call last): File "C:\Users\amata\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\amata\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['D:\\Kohya_GUI_Flux_Installer_30\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/Kohya_GUI_Flux_Installer_30/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'D:/Kohya_GUI_Flux_Installer_30/kohya_ss/output/shigeto_flux/model/config_lora-20240909-015530.toml']' returned non-zero exit status 1. 01:55:48-740726 INFO Training has ended.

Michael

Thank you! You were exactly right. I accidentally filled out the dreambooth portion before realizing my mistake. I didn't think it would break it xD. Also had to redownload the config file because I saved it along the way in dreambooth.

Michael

Hi, I just have a general curiosity question. How come the loras we are training are around 2.5 gb? I see many fluxD ones that are as low as 25 mb and produce very good results. Is there a way to "compress" our loras without quality loss?

SECourses

the only compression that wont cause quality loss is saving as fp16 - minimal loss. after that you have to reduce network lora rank which will reduce quality

SECourses

reduce VAE batch size to 1 if it is more than 1. also it is very likely that you at least 1 time loaded config into dreambooth tab and corrupted it forever

shigeto

Thank you. After re-installing and re-running from the beginning, this error no longer occurred, but the following error occurred File "C:\Users\amata\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\amata\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "D:\Kohya_GUI_Flux_Installer_30\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['D:\\Kohya_GUI_Flux_Installer_30\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/Kohya_GUI_Flux_Installer_30/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'D:/Kohya_GUI_Flux_Installer_30/kohya_ss/outputs/shigeto_flux/model/config_lora-20240909-104046.toml']' returned non-zero exit status 3221225477. 10:42:30-927356 INFO Training has ended.

SECourses

did you set accelerate? that error doesnt tell what is it the error

shigeto

I have downloaded it again since then, watched the youtube tutorial carefully from the beginning, and performed the steps, but I still get the following error. (never used dreambooth tab) One thing I am wondering is that during install, I was getting an error that xformers could not be installed, is this related? raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['D:\\Kohya_GUI_Flux_Installer_30\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/Kohya_GUI_Flux_Installer_30/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'D:/Kohya_GUI_Flux_Installer_30/kohya_ss/outputs/shigeto_flux/model/config_lora-20240909-184210.toml']' returned non-zero exit status 3221225477.

SECourses

no not related to xformers. second problem could be your RAM. how much you have? Also enable virtual RAM and set to 60 gb restart and try again

shigeto

I use RTX4090. Lora learning in sdxl was working fine (in main branch).

SECourses

that is GPU but your system RAM how much you have?

pratham

hi, the tutorial has been very much helpful as a complete beginner, but in the end I'm still getting an error, i spent around a whole day trying to find out what might be wrong but it seems like i have hit a dead end. how do i fix this? here's what im getting in my cmd 16:19:53-972928 INFO headless: False 16:19:54-002440 INFO Using shell=True when running external commands... Running on local URL: http://127.0.0.1:7860 To create a public link, set `share=True` in `launch()`. 16:20:27-764954 INFO Loading config... 16:22:24-261564 INFO Start training LoRA Flux1 ... 16:22:24-262565 INFO Validating lr scheduler arguments... 16:22:24-263566 INFO Validating optimizer arguments... 16:22:24-264566 INFO Validating C:\trained_images\log existence and writability... SUCCESS 16:22:24-264566 INFO Validating C:\trained_images\model existence and writability... SUCCESS 16:22:24-265564 INFO Validating C:/Kohya_GUI_Flux_Installer_23/flux_files/flux1-dev.safetensors existence... SUCCESS 16:22:24-266566 INFO Validating C:\trained_images\img existence... SUCCESS 16:22:24-267564 INFO Folder 1_ohwx man: 1 repeats found 16:22:24-268564 INFO Folder 1_ohwx man: 28 images found 16:22:24-268564 INFO Folder 1_ohwx man: 28 * 1 = 28 steps 16:22:24-269566 INFO Regulatization factor: 1 16:22:24-269566 INFO Total steps: 28 16:22:24-270566 INFO Train batch size: 1 16:22:24-271564 INFO Gradient accumulation steps: 1 16:22:24-271564 INFO Epoch: 200 16:22:24-272563 INFO max_train_steps (28 / 1 / 1 * 200 * 1) = 5600 16:22:24-272563 INFO stop_text_encoder_training = 0 16:22:24-273566 INFO lr_warmup_steps = 0 16:22:24-275566 INFO Saving training config to C:\trained_images\model\Best_v2_20240909-162224.json... 16:22:24-276566 INFO Executing command: C:\Kohya_GUI_Flux_Installer_23\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no --dynamo_mode default --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 C:/Kohya_GUI_Flux_Installer_23/kohya_ss/sd-scripts/flux_train_network.py --config_file C:\trained_images\model/config_lora-20240909-162224.toml C:\Kohya_GUI_Flux_Installer_23\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( C:\Kohya_GUI_Flux_Installer_23\kohya_ss\venv\lib\site-packages\xformers\ops\fmha\flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch. @torch.library.impl_abstract("xformers_flash::flash_fwd") C:\Kohya_GUI_Flux_Installer_23\kohya_ss\venv\lib\site-packages\xformers\ops\fmha\flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch. @torch.library.impl_abstract("xformers_flash::flash_bwd") C:\Kohya_GUI_Flux_Installer_23\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( 2024-09-09 16:22:32 INFO Loading settings from train_util.py:4189 C:\trained_images\model/config_lora-20240909-162224.toml... INFO C:\trained_images\model/config_lora-20240909-162224 train_util.py:4208 2024-09-09 16:22:32 INFO t5xxl_max_token_length: 512 flux_train_network.py:144 C:\Kohya_GUI_Flux_Installer_23\kohya_ss\venv\lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884 warnings.warn( You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 2024-09-09 16:22:33 INFO Using DreamBooth method. train_network.py:281 INFO prepare images. train_util.py:1803 INFO get image size from name of cache files train_util.py:1741 100%|████████████████████████████████████████████████████████████████████████████████| 28/28 [00:00<00:00, 1399.78it/s] INFO set image size from cache files: 28/28 train_util.py:1748 INFO found directory C:\trained_images\img\1_ohwx man contains 28 image files train_util.py:1750 INFO 28 train images with repeating. train_util.py:1844 INFO 0 reg images. train_util.py:1847 WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:1852 INFO [Dataset 0] config_util.py:570 batch_size: 1 resolution: (1024, 1024) enable_bucket: False network_multiplier: 1.0 [Subset 0 of Dataset 0] image_dir: "C:\trained_images\img\1_ohwx man" image_count: 28 num_repeats: 1 shuffle_caption: False keep_tokens: 0 keep_tokens_separator: caption_separator: , secondary_separator: None enable_wildcard: False caption_dropout_rate: 0.0 caption_dropout_every_n_epoches: 0 caption_tag_dropout_rate: 0.0 caption_prefix: None caption_suffix: None color_aug: False flip_aug: False face_crop_aug_range: None random_crop: False token_warmup_min: 1, token_warmup_step: 0, alpha_mask: False, is_reg: False class_tokens: ohwx man caption_extension: .txt INFO [Dataset 0] config_util.py:576 INFO loading image sizes. train_util.py:876 100%|██████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:00 flux_utils.py:55 INFO prepare split model flux_train_network.py:99 2024-09-09 16:22:34 INFO load state dict for lower flux_train_network.py:106 INFO load state dict for upper flux_train_network.py:111 INFO prepare upper model flux_train_network.py:114 16:22:42-596679 INFO Training has ended.

SECourses

very likely out of RAM. setup virtual RAM. also what is your GPU? If you upgrade to Gold membership i can connect your pc and fix hopefully

shigeto

Finally I could run this script. I'm not entirely sure, but it seems to have worked with the following turned off Cache Text Encoder Outputs Cache Text Encoder Outputs to Disk Memory Efficient Save

GrafT

HI, When i try to download the link:Kohya_GUI_Flux_Installer_30.zip,it shows that { "errors": [ { "code": 902, "code_name": "AttachmentNotFound", "detail": "Attachment with id 20420751 was not found.", "id": "755de518-de0b-513d-b721-4f07529ac096", "status": "404", "title": "Attachment was not found." } ] } how can i deal with it, thank you.

SECourses

refresh page it works at the moment

SECourses

because you are not using our configs. you enabled T5 training.

shigeto

You mean Train T5-XXL model check box? I am using your Rank_3_18950MB_9_05_Second_IT.json, so this was off all along.

SECourses

yes but it must be on otherwise you dont have to turn off Cache Text Encoder Outputs, Cache Text Encoder Outputs to Disk unless you have some other problem for some reason

shigeto

Since I unchecked the following three checkboxes at once, I am not sure which one is responsible for the error, so I will verify each one once I finish the Lora study that is running now. Cache Text Encoder Outputs Cache Text Encoder Outputs to Disk Memory Efficient Save Anyway, I am glad that the Lora training is running. Thank you so much!

gawdman

Hi Furkan, Great tutorials. I've been able to follow you and use your configs to do some trainings with good results with the Flux1.dev model. However, I'm curious if you looked into seeing training on the quantized models. The reason is that the LoRAs trained on the Flux1.dev model are OK with the full model, but they perform so-so on the quantized models (which makes sense). I can run the full model on RunPod and MassedCompute just fine, but locally, I obviously can't load the entire Flux1.dev model without something like 32GB minimum VRAM, but I can run fp8 or schnell. Have you tried training LoRAs on those models?

SECourses

Fp8 works perfect with loras so use fp8 + swarmui

SECourses

Kohya can't train quant model, not that I know any trainer can train.

gawdman

Sounds good. Unfortunately, I see that even fp8 has some slight degradation compared to full model, but it's definitely better than schnell. Thanks!

wylason

This guide is great! have gotten some great results with just a few images.... blows SDXL/Dreambooth out of water for both consistency and likeness.... One question though... I've done some trainings on my dog (using 'dog' as the class name) and while it seems to get general likeness and features down.... the color/coat of the dog is completely off (chooses color randomly, while my dog has black fur)... any ideas? Does it have to do with the Flux model itself just not being great with animals/dogs?

SECourses

in that case also add black fur to your prompts. in my case i add eyeglasses to get more likeliness :D and yes flux is next level

wylason

lol good call! any idea what may be causing the issue on the training/Lora side?

SECourses

when training nothing happens it internally fully captions , thus it removes my eyeglasses as well if i dont mention it

Rogério Bernardes

Hello! First of all, thank you so much for the guide. It’s been fantastic, and I've learned a lot from you. I’ve just created my first LoRA — I had never made one before, and Flux seems amazing for this purpose. I'm quite happy with the result, but I noticed that when I use it, the images have faint stripes in the lower-central area. The images in the dataset were of good quality and sized 1024x1024. Could this issue be related to the quality of the dataset? Could anyone help me with this? I'll include some sample images for reference. Dataset -> https://imgur.com/gallery/pOGRXed My Images with LoRA -> https://imgur.com/gallery/faint-stripes-lora-rWwlbjV What could be causing this? Thank you in advance!

SECourses

can you repeat training on massed compute and train in fp16 and compare? did you train on computer? which rank? i am suspecting this

Rogério Bernardes

I trained on Massed Compute; it wasn't done locally. I used 2 RTX A6000 GPUs and set the rank to 1 (for 1 GPU).

SECourses

i just checked your images again and they link to same place. you can upload your training images to another link?

Rogério Bernardes

Thank you for letting me know; I hadn’t noticed. I’ve just updated the link. You can now view the dataset and the results generated with LoRA!

SECourses

ye stripes are so obvious. i use SwarmUI, 16 bit precision and ipndm sampler. can you test with same settings? also 40 steps. and what resolution you generate images?=

The Room

How Pausing and resuming LORA training with kohya? Tutorial? Best regards

Sonivas Sx

Do we really need 1:1 512x512 Aspect ratio of the Dataset images when training a character ? or can we have different Dimensions in the same Dataset? are 2:1 ok, or do they yield worst results ?

SECourses

1:1 works best. So have a model that, have baseline then you can try multi aspect ratio resolution

Sonivas Sx

ill try that tonight cheers

TriVectorX

Hi Furkan, thank you very much for the great tutorial! When I start the training, it stops at the step of caching latents 0/15, do you have any idea of how to fix this? I'm training on my windows PC with 4090 and I have 128GB of RAM.

TriVectorX

Oh I see, thank you so much I will definitely watch that one and try again later. The chosen config file is Rank_3_18950MB_9_05_Second_IT, as I understand 4090 can handle this.

TriVectorX

Hi Furkan, thanks for the fix, it is working now! However, I see the class_tokens: ohwx_man (with a underscore between ohwx and man) instead of class_tokens: ohwx man (which only has space in between in your video tutorial). Did I do anything wrong here?

SECourses

if your folder name is 1_ohwx man then it is okay you can ignore it

SECourses

you can use rank 4 to speed up huge. other than that you have to reduce resolution which degrades quality a lot

SECourses

download v33 and use upgrade now it will install torch 2.5

Victor Mustin

Incredible! What would be some guidelines on what to change if our dataset is smaller ?

SECourses

well these entire research was done on 15 images dataset. or you mean even smaller? it is already shared in the post you can see.

sonofabear

Hello, I've tried using your settings to train a LoRa 2 different times and I keep getting images that look off (https://postimg.cc/nCV44Cq2). The likeness is there, but you can see by the image that the quality is bad, and the background is way to blurred. Without using the LoRa, the images look great. What am I doing wrong?

SECourses

no it wouldnt matter much. ohwx is still a rare token for clip l and possibly T5 XXL. so it is safe to use

SECourses

are you using swarmui and the settings i used in video after training the lora when generating images? if not please try my config and see if makes improvements . i couldnt open your image you can email me : monstermmorpg@gmail.com

Victor Mustin

I was referring to the 256 dataset training, but I have to admit I can't really sort through all these links. But yes say I have 15 images, which is the config that I should use ? The one that maximises likeness

SECourses

which GPU you have i can tell you according to that. your GPU and its VRAM

Victor Mustin

I train on runpod H100 with 80GB. I don't mind the cost I'm just looking for highest likeness, I also don't mind slightly overfitting. Thanks :)

SECourses

very nice then use Rank_1_T5_XXL_39200MB_9_46_Second_IT.json

Oinksauce

Ok to ask questions here? I'm curious why the LORA's produced here are so large? I'm getting 4.6GB for these, but LORA's on CIVITAI are typically 10-100mb in size. Is there a way to make them smaller without sacrificing quality?

SECourses

you can save as FP16, it will reduce size to half, other than that no way to reduce size without sacrificing quality. they are big since Clip L + T5 XXL also added to the LoRA and LoRA rank is 128 and we save as Float max quality :)

100Sachen80

With rank9 file, my loras are about 500mb in size, is that correct? seems a bit small? Also, could i reduce the resolution of the source images to 512px and set this in the training settings? would it be faster in training and use less vram? I have a 3060 12GB card.

pratham

You mean the system RAM? I have got 32gb of ram and also using an RTX 4070tiSuper, I will try resolving this on my own based on your reply but if it fails i will try upgrading to gold :)

SECourses

I have shown virtual ram in video check that part

pratham

I tried with virtual memory. around 92GB of it which i read was the recommended max limit and i have a Samsung 990pro SSD. But the outcome seems to be still the same. the moment it gets to loading "cache text encoder outputs" and the training ends. tracking the memory usage while i start training still appears to behave the same way. goes until 100% and drops down to 70% and then stops. is there anything else i can do to solve this ?

Oinksauce

hmm - Using the 4x GPU SLOW profile, the training keeps hanging at "move encoders to gpu" Using v33. I've tried 3x and on two different Pods on Runpod.

SECourses

did you try other configs? i did the multi gpu training on massed compute. did you set gpu ids? since it doesnt give error i dont know. if you upgrade gold membership i can connect your pc and try to help

SECourses

yes 512 would be faster but lower quality. the size depends on saving as float or fp16, also training clip l and t5 increases ize. moreover size also depends on network lora rank.

Oinksauce

I did not try other configs - will try another rank 1 config and see if that works. TY

SECourses

you are welcome let me know results

Bobbie

Take a look at the overall cloudflared tunnel feature to create a bridge rather than the gradio share links. If you need to move files around you can temporarily zip them up and put them in one of the output folders and access them via the tunnel. It also supports much better security options

GhostDance

Hey Furkan, Apologies if this has already been addressed, but during the installation step, I encountered an error that says: ERROR: Could not find a version that satisfies the requirement xformers==0.0.27.post2 (from versions: 0.0.28.post1) ERROR: No matching distribution found for xformers==0.0.27.post2 Do you have any insight on how to fix this?

SECourses

not important we dont use xformers. also on windows use Install_Torch_2_5_Dev_Huge_Speed_Up.bat it will uninstall xformers and upgrade torch to much faster working version

Oinksauce

In total, I tried the 4x FAST, 4x SLOW and the Rank_1_T5_XXL_39200MB_9_46_Second_IT profiles. The GPU id's were set on the 4x profiles and I set it on the third. The two 4x profiles froze after "moving encoders to GPU" and the third froze before that point. I didn't have time to investigate, but I didn't have luck with any today.

JASH MEHTA

hey furkan thanks for the tutorial, I have setup everything correctly but when the training starts it is just getting stuck, what should I do? I have a RTX 4090, also I am using rank 4 config Training has ended. 15:08:48-651407 INFO Start training Dreambooth... 15:08:48-652411 INFO Validating lr scheduler arguments... 15:08:48-652411 INFO Validating optimizer arguments... 15:08:48-653448 INFO Validating C:/Forge/webui_forge_cu121_torch21/webui/models/Stable-diffusion/jash-flux/log existence and writability... SUCCESS 15:08:48-654448 INFO Validating C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\model existence and writability... SUCCESS 15:08:48-654448 INFO Validating C:/Users/admin/Downloads/flux_dev.safetensors existence... SUCCESS 15:08:48-655448 INFO Validating C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\img existence... SUCCESS 15:08:48-656447 INFO Folder 1_ohwx man: 1 repeats found 15:08:48-657447 INFO Folder 1_ohwx man: 15 images found 15:08:48-657447 INFO Folder 1_ohwx man: 15 * 1 = 15 steps 15:08:48-658447 INFO Regulatization factor: 1 15:08:48-658447 INFO Total steps: 15 15:08:48-659446 INFO Train batch size: 1 15:08:48-659446 INFO Gradient accumulation steps: 1 15:08:48-660448 INFO Epoch: 200 15:08:48-660448 INFO max_train_steps (15 / 1 / 1 * 200 * 1) = 3000 15:08:48-661448 INFO lr_warmup_steps = 0 15:08:48-662593 INFO Saving training config to C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\model\jash_flux_20240916-150848 .json... 15:08:48-663460 INFO Executing command: C:\Kohya_GUI_Flux_Installer_v33\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no --dynamo_mode default --gpu_ids 0 --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 C:/Kohya_GUI_Flux_Installer_v33/kohya_ss/sd-scripts/flux_train.py --config_file C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\model/config_dreambooth-2024091 6-150848.toml C:\Kohya_GUI_Flux_Installer_v33\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( C:\Kohya_GUI_Flux_Installer_v33\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( C:\Kohya_GUI_Flux_Installer_v33\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( 2024-09-16 15:08:57 INFO Loading settings from train_util.py:4230 C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\model/co nfig_dreambooth-20240916-150848.toml... INFO C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\model/co train_util.py:4249 nfig_dreambooth-20240916-150848 2024-09-16 15:08:57 INFO Using DreamBooth method. flux_train.py:101 INFO prepare images. train_util.py:1807 INFO get image size from name of cache files train_util.py:1745 100%|███████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 2137.19it/s] INFO set image size from cache files: 15/15 train_util.py:1752 INFO found directory train_util.py:1754 C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\img\1_oh wx man contains 15 image files WARNING No caption file found for 15 images. Training will continue without captions for train_util.py:1785 these images. If class token exists, it will be used. / 15枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャ プションなしで学習を続行します。class tokenが存在する場合はそれを使います。 WARNING C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\img\1_oh train_util.py:1792 wx man\a_photo_of_ohwxman (1).JPG WARNING C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\img\1_oh train_util.py:1792 wx man\a_photo_of_ohwxman (10).JPG WARNING C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\img\1_oh train_util.py:1792 wx man\a_photo_of_ohwxman (11).JPG WARNING C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\img\1_oh train_util.py:1792 wx man\a_photo_of_ohwxman (12).JPG WARNING C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\img\1_oh train_util.py:1792 wx man\a_photo_of_ohwxman (13).JPG WARNING C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\img\1_oh train_util.py:1790 wx man\a_photo_of_ohwxman (14).JPG... and 10 more INFO 15 train images with repeating. train_util.py:1848 INFO 0 reg images. train_util.py:1851 WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:1856 INFO [Dataset 0] config_util.py:570 batch_size: 1 resolution: (1024, 1024) enable_bucket: False network_multiplier: 1.0 [Subset 0 of Dataset 0] image_dir: "C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\img\1_o hwx man" image_count: 15 num_repeats: 1 shuffle_caption: False keep_tokens: 0 keep_tokens_separator: caption_separator: , secondary_separator: None enable_wildcard: False caption_dropout_rate: 0 caption_dropout_every_n_epoches: 0 caption_tag_dropout_rate: 0.0 caption_prefix: None caption_suffix: None color_aug: False flip_aug: False face_crop_aug_range: None random_crop: False token_warmup_min: 1, token_warmup_step: 0, alpha_mask: False, is_reg: False class_tokens: ohwx man caption_extension: .txt INFO [Dataset 0] config_util.py:576 INFO loading image sizes. train_util.py:880 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00 flux_utils.py:69 INFO [Dataset 0] train_util.py:2328 INFO caching latents with caching strategy. train_util.py:988 INFO checking cache validity... train_util.py:998 100%|███████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 2246.87it/s] INFO no latents to cache train_util.py:1038 C:\Kohya_GUI_Flux_Installer_v33\kohya_ss\venv\lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884 warnings.warn( You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 2024-09-16 15:08:58 INFO Building CLIP flux_utils.py:74 INFO Loading state dict from flux_utils.py:167 C:\ComfyUI_windows_portable\ComfyUI\models\clip\clip_l.safetensors INFO Loaded CLIP: flux_utils.py:170 INFO Loading state dict from flux_utils.py:215 C:\ComfyUI_windows_portable\ComfyUI\models\clip\t5xxl_fp16.safetensors INFO Loaded T5xxl: flux_utils.py:218 2024-09-16 15:09:13 INFO [Dataset 0] train_util.py:2349 INFO caching Text Encoder outputs with caching strategy. train_util.py:1111 INFO checking cache validity... train_util.py:1117 100%|███████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 2295.40it/s] INFO no Text Encoder outputs to cache train_util.py:1139 INFO cache Text Encoder outputs for sample prompt: flux_train.py:235 C:\Forge\webui_forge_cu121_torch21\webui\models\Stable-diffusion\jash-flux\model\sam ple/prompt.txt INFO Building Flux model dev flux_utils.py:45 INFO Loading state dict from C:/Users/admin/Downloads/flux_dev.safetensors flux_utils.py:52 INFO Loaded Flux: flux_utils.py:55 FLUX: Gradient checkpointing enabled. CPU offload: False INFO enable block swap: double_blocks_to_swap=0, single_blocks_to_swap=0 flux_train.py:272 number of trainable parameters: 11901408320 prepare optimizer, data loader etc. INFO use Adafactor optimizer | {'scale_parameter': False, 'relative_step': False, train_util.py:4541 'warmup_init': False, 'weight_decay': 0.01} WARNING because max_grad_norm is set, clip_grad_norm is enabled. consider set to 0 / train_util.py:4569 max_grad_normが設定されているためclip_grad_normが有効になります。0に設定して無効に したほうがいいかもしれません WARNING constant_with_warmup will be good / train_util.py:4573 スケジューラはconstant_with_warmupが良いかもしれません enable full bf16 training. running training / 学習開始 num examples / サンプル数: 15 num batches per epoch / 1epochのバッチ数: 15 num epochs / epoch数: 200 batch size per device / バッチサイズ: 1 gradient accumulation steps / 勾配を合計するステップ数 = 1 total optimization steps / 学習ステップ数: 3000 steps: 0%| | 0/3000 [00:00 2024-09-16 15:09:33 INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:672

SECourses

email me entire logs this doesnt tell the error : monstermmorpg@gmail.com

SECourses

you need to upgrade 2.5 use my bat file, it speeds up training

SECourses

sadly hard to debug with this info

SECourses

Download latest zip file and it has another bat file look all names

Manpreet Singh

What was the impact text encoder training on vs off? If you got a chance to compare.

SECourses

Clip L text encoder improves significantly. T5 is very minimal impact in my tests.

SECourses

we dont use xformers during training

Judi Godvliet

You are a true master. I am training right now my portrait with Lora using Flux and 20 photos with 200 epoch. I train locally using my rtx 4090 gfxcard. Is it normal that the Lora training will take 4 a 5 hours or did I miss an optimization script or setting?

SECourses

first thank you. duration sounds to me right. what is your step speed like how many seconds it and also which rank?

SECourses

ye that is a good speed. linux faster but this is what we have on windows. it is like 100% faster than rtx 3090

Sasha

hello Furkan, im dealing with this error second day, need help. i have 3070 ti (laptop). 00:35:29-062276 INFO Copy C:/PavelGoliksteal to C:/FluxTraining\img/1_ohwx style... 00:35:29-093924 INFO Regularization images directory is missing... not copying regularisation images... 00:35:29-095832 INFO Done creating kohya_ss training folder structure at C:/FluxTraining... 00:35:55-757651 INFO Start training LoRA Flux1 ... 00:35:55-758655 INFO Validating lr scheduler arguments... 00:35:55-759657 INFO Validating optimizer arguments... 00:35:55-760659 INFO Validating lora type is Flux1 if flux1 checkbox is checked... 00:35:55-761662 INFO Validating C:/FluxTraining\log existence and writability... SUCCESS 00:35:55-763667 INFO Validating C:/FluxTraining\model existence and writability... SUCCESS 00:35:55-764670 INFO Validating C:/WORK/Apps/Kohya/flux1-dev.safetensors existence... SUCCESS 00:35:55-765673 INFO Validating C:/FluxTraining\img existence... SUCCESS 00:35:55-766675 INFO Folder 1_ohwx style: 1 repeats found 00:35:55-767679 INFO Folder 1_ohwx style: 15 images found 00:35:55-768680 INFO Folder 1_ohwx style: 15 * 1 = 15 steps 00:35:55-769683 INFO Regulatization factor: 1 00:35:55-770686 INFO Train batch size: 1 00:35:55-772169 INFO Gradient accumulation steps: 1 00:35:55-772673 INFO Epoch: 150 00:35:55-773672 INFO max_train_steps (15 / 1 / 1 * 150 * 1) = 2250 00:35:55-774675 INFO stop_text_encoder_training = 0 00:35:55-779166 INFO lr_warmup_steps = 0 00:35:55-782174 INFO Saving training config to C:/FluxTraining\model\FirstTryGolik_20240919-003555.json... 00:35:55-784179 INFO Executing command: C:\WORK\Apps\Kohya\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no --dynamo_mode default --gpu_ids 0 --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 C:/WORK/Apps/Kohya/kohya_ss/sd-scripts/flux_train_network.py --config_file C:/FluxTraining\model/config_lora-20240919-003555.toml 2024-09-19 00:36:03 WARNING WARNING[XFORMERS]: xFormers can't load C++/CUDA _cpp_lib.py:148 extensions. xFormers was built for: PyTorch 2.4.0+cu121 with CUDA 1201 (you have 2.0.1+cu117) Python 3.10.11 (you have 3.10.11) Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-x formers) Memory-efficient attention, SwiGLU, sparse and more won't be available. Set XFORMERS_MORE_DETAILS=1 for more details Traceback (most recent call last): File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\xformers\checkpoint.py", line 53, in from torch.utils.checkpoint import SAC_IGNORED_OPS as _ignored_ops # type: ignore ImportError: cannot import name 'SAC_IGNORED_OPS' from 'torch.utils.checkpoint' (C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 710, in _get_module return importlib.import_module("." + module_name, self.__name__) File "C:\Users\sasab\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\loaders\ip_adapter.py", line 34, in from ..models.attention_processor import ( File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\models\attention_processor.py", line 31, in import xformers File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\xformers\__init__.py", line 12, in from .checkpoint import ( # noqa: E402, F401 File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\xformers\checkpoint.py", line 57, in from torch.utils.checkpoint import _ignored_ops # type: ignore ImportError: cannot import name '_ignored_ops' from 'torch.utils.checkpoint' (C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 710, in _get_module return importlib.import_module("." + module_name, self.__name__) File "C:\Users\sasab\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion.py", line 24, in from ...loaders import FromSingleFileMixin, IPAdapterMixin, LoraLoaderMixin, TextualInversionLoaderMixin File "", line 1075, in _handle_fromlist File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 700, in __getattr__ module = self._get_module(self._class_to_module[name]) File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 712, in _get_module raise RuntimeError( RuntimeError: Failed to import diffusers.loaders.ip_adapter because of the following error (look up to see its traceback): cannot import name '_ignored_ops' from 'torch.utils.checkpoint' (C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\WORK\Apps\Kohya\kohya_ss\sd-scripts\flux_train_network.py", line 13, in from library import flux_models, flux_train_utils, flux_utils, sd3_train_utils, strategy_base, strategy_flux, train_util File "C:\WORK\Apps\Kohya\kohya_ss\sd-scripts\library\flux_train_utils.py", line 17, in from library import flux_models, flux_utils, strategy_base, train_util File "C:\WORK\Apps\Kohya\kohya_ss\sd-scripts\library\train_util.py", line 52, in from diffusers import ( File "", line 1075, in _handle_fromlist File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 701, in __getattr__ value = getattr(module, name) File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 701, in __getattr__ value = getattr(module, name) File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 700, in __getattr__ module = self._get_module(self._class_to_module[name]) File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 712, in _get_module raise RuntimeError( RuntimeError: Failed to import diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion because of the following error (look up to see its traceback): Failed to import diffusers.loaders.ip_adapter because of the following error (look up to see its traceback): cannot import name '_ignored_ops' from 'torch.utils.checkpoint' (C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py) Traceback (most recent call last): File "C:\Users\sasab\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\sasab\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\WORK\Apps\Kohya\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "C:\WORK\Apps\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\\WORK\\Apps\\Kohya\\kohya_ss\\venv\\Scripts\\python.exe', 'C:/WORK/Apps/Kohya/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'C:/FluxTraining\\model/config_lora-20240919-003555.toml']' returned non-zero exit status 1. 00:36:05-159564 INFO Training has ended

SECourses

please download latest zip file and run torch 2.5, it will uninstall xformers and dont select xformers use configs as they are

puk

Hello Furkan, first of all thank you for your incredible work and passion to teach us so much about Lora Training. I am diligently following and trying out your progress with FLUX training. So far my results have been very good. I started with 50 photos and then 100 photos. Finally I have now trained 3,400 photos as I did with SDXL. I train everything locally on my 4090. the training with 3,400 photos took almost 40 hours with 1 repeat and 10 epochs. The results with 3,400 are really good. But what I have noticed. I don't get the elements generated that I got with SDXL. I'll explain briefly what my goal is. I have 3,400 photos of a person in different outfits, locations, poses and expressions. The goal is to end up with a Lora that can reproduce all these elements. With SDXL I have achieved this with captioning. You say that Flux doesn't need detailed captions, but I still did it with the captions from my SDXL set. When I enter them during generation I don't get the results I was hoping for. My question is, does the training in Flux work the same way as in SDXL? Which way would you choose so that I can generate all elements of the training photos? Translated with www.DeepL.com/Translator (free version)

SECourses

No it doesn't work same way as SDXL. Flux really tends to mix multi concepts and you are kind of trying that. We couldn't solve this issue yet sadly

The Room

Hi, I've created 20 LoRAs with a dataset of around 50,000 frames taken from the FMVs of a specific video game. However, since the process would have been very long, I divided them into 20 LoRAs. I'd like to know if there's a way to combine all 20 LoRAs into a single one. Or perhaps some other solution. Otherwise, I'll have to use all 20 of them together to generate images, and I don't know if there will be any conflicts. If it's possible to combine them, I'd be happy to hear it. The theme is always the same since we're talking about FMV of a specific video game, so in theory, it should work? And be compatible with each other? I only used the trigger word without a prompt since I'm interested in the specific style. Then flux does the rest. To create the text files very quickly, I created a Python script that creates a text file for each frame with the same name and my chosen trigger word, and automatically calculates the number of frames in the folder. Best regards

SECourses

Kohya GUI has LoRA merge feature but i doubt that it will work as you expect.

SECourses

use latest zip file it will uninstall xformers

Rolf Steiner

INFO move text encoders to gpu flux_train_network.py:216 Traceback (most recent call last): File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\sd-scripts\flux_train_network.py", line 519, in trainer.train(args) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\sd-scripts\train_network.py", line 402, in train self.cache_text_encoder_outputs_if_needed(args, accelerator, unet, vae, text_encoders, train_dataset_group, weight_dtype) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\sd-scripts\flux_train_network.py", line 218, in cache_text_encoder_outputs_if_needed text_encoders[1].to(accelerator.device) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\lib\site-packages\transformers\modeling_utils.py", line 2905, in to return super().to(*args, **kwargs) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1340, in to return self._apply(convert) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply module._apply(fn) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply module._apply(fn) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply module._apply(fn) [Previous line repeated 4 more times] File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 927, in _apply param_applied = fn(param) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1326, in convert return t.to( torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 80.00 MiB. GPU 0 has a total capacity of 16.00 GiB of which 10.91 GiB is free. Of the allocated memory 3.92 GiB is allocated by PyTorch, and 14.50 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) Traceback (most recent call last): File "C:\Users\fotog\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\fotog\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "D:\Kohya_GUI_Flux_Installer_v36\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['D:\\Kohya_GUI_Flux_Installer_v36\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/Kohya_GUI_Flux_Installer_v36/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'D:/Nová složka\\model/config_lora-20240921-105913.toml']' returned non-zero exit status 1. 11:00:18-302466 INFO Training has ended.

SECourses

your vram was not sufficient. either reduce your VRAM before starting to training or use lower VRAM. and i see your paths has special chars (složka) it is a big red flag for AI. try rank 5 it is slow but best and should work fine on 16 gb

Rolf Steiner

I tried to rename the folder and on rank 5 I'm trying it all the time. finally I changed the encoder from t5xxl_fp16 to t5xxl_fp8_e4m3fn and so far it seems to work

100Sachen80

Do i have to use t5 file and clip file and ae file in the corresponding folders and select them in swarmui when creating an image, or does ist automatically load? or are those files not needed for creation? it is better with those files, like the t5 file?

SECourses

you need to put clip and vae file. it auto downloads fp8 T5

100Sachen80

When using swarmui and flux bnb-v4 model, loras dont work. can you reproduce that? it always gives error [ComfyUI-0/STDERR] RuntimeError: .to() does not accept copy argument

SECourses

yes bnb not compatible dont waste time. i suggest use fp8 dev

100Sachen80

So Generation is better quality with Them? Because it also generates even when Not actively selecting and enabling them in the UI

SECourses

if files exists it will work. i would use T5 16 bit a little bit better, you need to either manually select or download with certain file name our downloaders download

SECourses

your rank 3 was probably corrupted due to loading into dreambooth tab :D 3.59 decent speed

Robert Arsene

I am trying to use some of your test prompts with photo of in SwarmUI. I think this is what makes full body generation with better faces, right? I only get a blurry image and only the face is clear.

SECourses

you must be doing something wrong. i am preparing new prompts and will show in more details hopefully in upcoming video

Robert Arsene

Seems the issue is I only get blurry images, even with other prompts.

Robert Arsene

Better now. Thanks. Still trying to get the quality face generation you get. It works well for close-up, but when the subject is further or full body, it barely works.

Robert Arsene

Any easy way to upscale them in SwarmUI for better quality? Could not find a specific tutorial for Flux.

Robert Arsene

SUPIR tends to distort or inadequately process faces when they are not the central element of the image or when they are situated deeper in the image's depth of field. Let's hope the new Flux ControlNet Upscale is good. Trying a workflow with it borrowed from a chinese guy.

SECourses

have you tried face enhance of the SUPIR it may help. both enable face and background enhance options

100Sachen80

When using multi GPU (2x GPU A6000) in Massed Compute, do I have to adjust the LR for Lora by your formula slightly, going from 0.00005 to 0.00006 or 0.00007 maybe? Your formula says that I should not adjust the LR, because LR = ( 0.00005 * 2 / 2 ) stays the same LR, but should it not also be adjusted slightly to take the dividing / splitting on 2x GPU into account? Thank you for all your work and all your help here!

JustLookingForNothin

I wonder if somebody could share (or point me to) a FLUX1.dev workflow for ComfyUI XY-plot to test the generated Loras with several epochs.

SECourses

why don't you try SwarmUI? its x/y/z plot system is amazing

SECourses

thank you so much for comment. it is a good question. if you have too many images i would say not increase. if you dont have many images like only 15 images you can increase as you said

Charles Leo

Going a little crazy as I do not have the checkbox for Flux as you do in your LORA video. It shows v2, v_parameterization, and SDXL only. Everything is up to date.

SECourses

when you select flux dev1 model it appears - select it from your pc

Charles Leo

Unfortunately it doesn’t. I even went to install it via Stability Matrix instead of installing Koyha_ss manually thinking the install didn’t go properly and even that version doesn’t provide the option.

SECourses

Stability Matrix wont work. you have to use our installer since we switch to accurate branch

Charles Leo

I also tried it manually directly from the latest repo prior to trying to install it via SM. I made sure to get the latest updates and even latest 2.5 torch. I took a screenshot but seems like this interface wont allow me to share it here.

Charles Leo

I’ll try giving it another clean install or on another pc. One thing I will say is that I tend to install everything manually first with latest updates and only then load in you config file. I prefer to know ahead of time what I’m installing with other bats and sources.

SECourses

you can edit my installer and see what it does so easy.

Charles Leo

Yes, but to save a little time are you adding a radio/checkbox for Flux independently that’s not part of the formal Koyha_SS commit? Is it a separate fork that can be found?

Charles Leo

Okay my oversight as I wasn't using your step 1 (especially if some of us already had Koyha_ss installed prior.) You're pulling/checkout a specific branch 'sd3-flux.1' which is probably the missing piece of the puzzle. I'm surprised that it's so many commits ahead of the Master (140 to be precise.) Also, following your video and writings here the order seems a bit mixed up as I watched half your video first PRIOR to even downloading your configs, not even realizing that you're trying to automate most of the installs for other people as I'm new here, but not to CG or AI in general. I'm more granular so by skipping, I missed this piece. Two other things I'd add is that your bats assume a specific install location. I prefer to designate my own directory structure/naming convention. The other is that I'd prefer to not have duplicate flux or other models floating around on the same drive and would rather share one file per different applications. It's one of the few reasons I started using Stability Matrix versus manual installations was for a more consistent organization of assets.

SECourses

ye i understand you but sadly it is impossible to cover everyone. so either you have to be experienced to figure out or follow me exactly :D

Charles Leo

Walking more through your installers. Two more things: 1. would you bother enabling NUMA? 2. I've seen this a few times and manually added it to my directory. Not sure what it does exactly or if it's even important: "python: can't open file 'C:\\Windows\\System32\\Fix_GUI_Bat.py'"

SECourses

1 i also dont know. 2 related to your folders. you are not following me. you need to follow tutorial.

Charles Leo

No offense, but you've contradicted yourself. In your video, you specifically state to look for errors. Upon following your installers, the output (I believe twice) says that this particular file is missing in System32. HOWEVER, it is in your install folder. So I manually added it to the System32 directory only to find out that it breaks the Koyha gui.bat. You may want to consider removing those lines calling for this file as it's possibly a leftover from maybe a previous script? I can't say for certain but it doesn't serve any purpose.

Charles Leo

I see what you're doing now and I'm frankly surprised there's that many new clueless people trying to generate bad manga at home. But hey you're trying to assist that denominator and I get it as well. I had hoped to circumvent hours of experimentations (as a background I've helped pioneer CUDA and rendering applications over 15 years ago which led to these AI developments) by following some of your tests. I know you're a busy person but it would be great to have a synopsis for more experienced users. I'll also admit that my experience is limited with training LORAs so I was hoping this would be a starter course. Mainly just looking for general settings and a conf. to work with. My big mistake was not realizing that there was a separate branch for Flux so stupid me.

Charles Leo

Yes. I'm well aware (see above) and that's where I manually moved it from. but your bats do NOT install that into the directory and your output says it's missing... What does that file do/why does it exist? Is it not reasonable for someone to want to know what's being installed or if it's even needed? You said to pay attention to errors, but you have haven't even addressed your own. You know, I defended you yesterday on Reddit but now I'm starting to see what others are saying after these interactions. You're tone is condescending and I'd appreciate it if you talked to me like you would want to be treated as well. You're not always going to be the smartest person in the room either. It's not much to ask for.

Charles Leo

Oh FTR, that py file that's a leftover/not even needed and throws out errors via the bat is setting the incorrect Python version... 3.10.9

SECourses

i really dont understand your questions because you really should join discord and show screenshots or video of what you are doing. impossible to know what you are doing from text what that file does is replacing kohya bat file so that when you start the application it will not return back to torch 2.4 from torch 2.5. torch 2.5 speeds up the training on windows a lot also if you need installation help i connect pc of gold supporters and help them to setup

SECourses

by the way sorry for my tone if i sounded arrogant. i am really tired at the moment.

Charles Leo

No worries and I appreciate it. We all have shit days and moments. I’ll see if I can grab some screens tomorrow and hop on discord.

SECourses

thank you for understanding. yes discord is best.

Charles Leo

What might happen if you were to use a different clip model such as t5xxl_fp8_e4m3fn?

SECourses

i think it is not suggested. base fp16 models are auto casted according to the settings so it doesnt matter during training

Louis Lee

Can train with Flux Schnell ?

SECourses

it appears you are using your own ubuntu installation. you need add sudo to your commands sudo apt-get install python3-tk and sudo apt-get install python3-tk also try solutions given by chat gpt : https://poe.com/s/SKnYvYG4V8ngB8oKCT11

Charles Leo

Thanks. If I get a spare moment I might give it a test run even if at lower epochs to see if it breaks. I found this clip by City96 to be better at handling prompts but unsure how. It handles here.

Charles Leo

Trying your Massed Compute method out and it seems to be hanging forever on initializing. It's been around 5 minutes now. How long does that typically take? Oh and apologies but dislike Discord with a fiery passion. Worse interface ever.

SECourses

5-10 minute usually. you dont like discord but we have top staff of massed compute in discord we can ping :D

Kun HUANG

I don't know what's the error means: enable full bf16 training. Traceback (most recent call last): File "D:\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "D:\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "D:\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "D:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "D:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "D:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['D:\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/kohya_ss/sd-scripts/flux_train.py', '--config_file', 'D:/kohya_ss/outputs/kun\\model/config_dreambooth-20241006-150903.toml', '--cpu_offload_checkpointing']' returned non-zero exit status 3221225477. 15:09:23-564854 INFO Training has ended.

SECourses

Very likely out of RAM error. how much RAM you have? setup 64 GB virtual ram it should fix it

jai

Hello! Thanks to you, I'm really getting a lot of help. I understand that quality is important, but it seems that the flux Lora of about 60-70mb shared on civitai also has pretty good quality. Even when generating with fp16, the size of over 1GB is troublesome. Is there any way to generate a Lora with a size of around 70mb like this? Or would it be impossible with koyha? https://www.reddit.com/r/StableDiffusion/comments/1f523bd/good_flux_loras_can_be_less_than_45mb_128_dim/

SECourses

sure just save as fp16 and reduce lora network dimension to 32 from 128, this will reduce size from 2gb to 250 MB

Kun HUANG

thanks, it really works when i extend the virtual ram!

Alex puliatti

hi, when i run the windows install step 1 and i enter 1 to install the gui the bat closes straight away?

SECourses

hi open a cmd window and type bat file name and run from there, so we can see error reason. very likely you are missing requirements : https://youtu.be/DrhUHnYfwC0 also dont run as administrator

cool1

I tried installing this Kohya (I have the other) and I ran the "install torch 2.5 dev huge speed up". then when trying to run gui.bat it says "the procedure entry point ?set_tracing@SAved TensorDefaultHooks... could not be located in the dynamic link library g:\Kohy\...site-packages\torchvision\_C.pyd then in the command window "operator torchvision::nms does not exist" What's the easiest fix? Should I re-install without that speed up .bat file (or will that be too slow at running)?

cool1

Thanks. I'll try going through that other video. I tried re-installing everything (including the torch 2.5 dev huge speed up" and when trying to run the gui.bat it said things like "WARNING Package wrong version: torch 2.5.0+cu124 required 2.4.1+cu124 09:01:34-225354 INFO Installing package: torch==2.4.1+cu124" So it seems like torch 2.5 (that' supposed to give the huge speed up) isn't compatible with it, and the requirements want a lower torch version.

SECourses

you have to use Windows_Start_Kohya_SS.bat it has skip args

cool1

Thanks. I tried that and it quickly exits with the same error messages (too fast to read but they look the same as the ones shown when using gui.bat). That's after re-installing and using the torch 2.5 speed up bat file before it. If I re-install and don't click the "torch 2.5 dev huge speed up" clicking on gui.bat works and loads and says it's running on a particular ip address (like the standard one). Though I assume it will be a lot slower than if torch 2.5 could work (but it doesn't seem compatible with it/the requirements).

SECourses

torch 2.5 works i don't know what you are doing :) open a cmd and run bat file from that cmd to see message

cool1

Thanks. Somehow it worked after clicking the "install torch 2.5 dev.. bat file again". I'd already done that before a few times but this time it worked and allowed me to run the start windows bat with it seeming to load okay (haven't tried gui.bat again yet).

cool1

I tried training with the "rank 3" option (with rtx3090). It was taking ages but it would have worked if I left it for days (though I hadn't changed it to save more often). Because it was running slow I tried changing the "rank 4" option. But that seems more likely to fail with an error of things like "...Kohya-GUI-Flux-trainer\kohya_ss\sd-scripts\library\flux_models.py", line 1024, in forward if img.ndim != 3 or txt.ndim != 3: AttributeError: 'NoneType' object has no attribute 'ndim' Maybe if it's training with lower precision, when outputting sample images maybe it might be more likely to generate one where one of the values isn't a valid number? Is that what that error message could mean? Could it be anything to do with me increasing network rank to 275 and changing network alpha to 68? Maybe I could have used the "rank 3" one and just increased the learning rate so it could learn faster (but maybe the quality might not be as good then/it would overtrain). I tried training higher res (1280x1280) but that's just too slow with 1 RTX3090, especially when outputting an image sample. The changed Kohya also isn't as good at saving configurations as the standard Kohya because in this verison there's no "save as" option to save a copy of a config in a different name. edit: I think that error might have been something to do with the caching prompts options, because it says you can edit the prompts.txt file that has the prompts used training samples, but if you do and it's cached the prompts to memory/disc it seems to give an error if it finds one that's changed in the prompt.txt file.

Leonidas Freud

Due to regional issues, I attempted to rent a GPU on an alternative platform. I'm using Jupyter Notebook, and since this platform is similar to RunPod, I tried to apply RunPod's configuration. However, I encountered numerous and varied errors. I'm unsure of their causes or how to fix them. Occasionally, I manage to solve some issues, but I don't fully understand how or why the solutions work. I have some basic knowledge of Python and Linux. What should I study to better equip myself for building an environment to train LoRA on this new platform? I'd appreciate any suggestions on resources or topics to focus on. My goal is to understand the process better and troubleshoot more effectively. Thank you in advance for your help!

SECourses

yes caching can cause that 100%

SECourses

hi. those errors are very likely to inaccurate python, cudnn, cuda libraries. you need to install accurate ones. python 3.10, cuda 12.4, cudnn 8.9.7 and you need c++ tools

Maso

I created a flux lora, which I thought was only working with flux-dev (fp16), however upon further investigation it appears to work with only flux-dev checkpoints, but only if I change the setting in Forge 'Diffusion in low bits' to 'Automatic (fp-16 LoRA). I just wondered if you had seen similar? I'm using Rank3 (non T5) and the only Kohya settings I have changed are network dimension 32 and save as fp16, to reduce the sizee.

SECourses

hello can you elaborate more because i couldn't exactly understand

Maso

I'll try, in Forge UI, if I try and use my lora with a model, it doesn't work, but if I change the 'Diffusion in Low Bits' option (top right) to from 'Automatic' to 'Automatic (fp16)' it works ok. I was just curious why as I've never had an issue with any other lora I've downloaded. https://imgur.com/a/XpLbCyj

V Santhosh

Hi there, I'm using RTX 4090, I've enabled all the options like you've showed in the tutorial, When I hit start training I get a eerror saying T5Xxl is already training and cannot be used with cache text encoder outputs or something. So I disabled T5xxl training and started it. What do I lose because of this? Also for a 4090, If I use rank 3 24gb version config file. It's taking around 6-7 seconds per iteration. Which I think is very slow.

SECourses

Rank 3 is slow. if you want speed you need rank 4. but i suggest you to do fine tuning instead of lora. it is way better and almost same 6 7 second speed. i am gonna publish comparison grids hopefully tomorrow : https://www.patreon.com/posts/112099700 fine tuning way way better than lora

V Santhosh

Is there a tutorial video on it, on your youtube? I'm not great at following written instructions. A video would help a lot. Thanks

ole asdsad

Hey, everything works perfectly and my flux lora works great. I'm now wondering if i can use the lora model with another checkpoint like epiCRealism? Should i train a lora model specifically for that or can i somehow use the same lora model to keep my character's face consistent?

SECourses

best case is training on each model. but you can use lora trained on sd 1.5 models on different sd 1.5 models, sdxl on sdxl models and flux on flux models. sd 1.5 vs sdxl vs flux won't work. they only work on same base models

Zabi G

After clicking on Installer_bat_step1, it is showing this error.. ERROR: Cannot install -r requirements.txt (line 1), -r requirements.txt (line 10), diffusers[torch]==0.25.0 and huggingface-hub==0.24.5 because these package versions have conflicting dependencies.

Pixel Reaction

Massed Compute / DreamBooth INFO caching latents with caching strategy. INFO caching latents... 0%| Traceback (most recent call last): File "/home/Ubuntu/apps/kohya_ss/venv/bin/accelerate", line 8, in sys.exit(main()) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main args.func(args) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command simple_launcher(args) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/home/Ubuntu/apps/kohya_ss/venv/bin/python', '/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py', '--config_file', '/home/Ubuntu/Downloads/training_imgs/model/config_dreambooth-20241015-224004.toml']' died with . Training has ended.

SECourses

out of RAM error. don't use 24 GB RAM configs which are alt config and spot instance

Icaro Diniz

Hi First of all, congratulations on the amazing work! I would like to know where I can find the tutorial for Flux fine-tuning/Dreambooth, please.

SECourses

hopefully will be published once i edited. editing the tutorial

Federico Salmaso

Hi all, is anybody facing the same problem? when I click Windows install step 1, then I pick option 1, the process and windows close immediately. I have installed phyton same version, git, and cuda. thanks in advance

SECourses

what is your step speed? yes 256 images takes more time but certainly not forever on rtx 4090 :D

SECourses

please follow this video exactly as it is and then reinstall : https://youtu.be/DrhUHnYfwC0 also start bat file as this. open a cmd and type full name of the bat file and run that way to see error reason

Twinklixx Jones

Hello there! Do you have information on how to install flux into Kohya without using your .zip file? Your Windows_Install_Step_1.bat file does not account for PCs with multiple python versions installed, thus I can't use your installation method. This is likely the same problem that the person above me is having.

Twinklixx Jones

I can get "Windows_Update_Kohya_and_Fix_FLUX_Step2" to work on a separate install of kohya, however it states that everything is already up to date, but then when loading Flux Dev in Kohya I don't get the Flux or SD3 Checkbox. Edit: Eh I just uninstalled my other version of python and will reinstall later lol.

대훈 조

When I try to install Kohya_GUI_Flux_Installer_v41.zip, it says a virus was detected, and the download is blocked. I am using the Google Chrome browser. Is there a solution for this?

Steve

What is a good way to transfer large files 23GB + 23GB saved training state to Massed compute? Transfer speed is very slow. Thanks!

Steve

Hello, in massed compute Kohya Flux Dreambooth training I am getting 2024-10-27 15:19:31 INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715 Could not load library libnvrtc.so.12. Error: libnvrtc.so.12: cannot open shared object file: No such file or directory I assume this has to do with CUDA. Do you know how to resolve it?

SECourses

let me make a fresh install and test

SECourses

hi which config did you try? it is out of vram error. are you also generating samples during training?

SECourses

ye that one barely fitting into 24 gb so i see. in your case it did few steps and got error later. so you need to use either bigger GPU or use the one without T5 XXL

SECourses

i see it is working so i think you can ignore. i see speed is also good

SECourses

you are welcome. well i tested prodigy in past and i didnt find it better than adafactor personally. since I find a very good LR for batch size 1 and each batch size, i think our config works good. we use static learning rate (LR)

Steve

Hi, were you able to find the issue? Ty

Steve

Hello I tried again, it is running now but still gives the error. Is this ok? epoch 1/100 2024-11-01 20:35:23 INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715 Could not load library libnvrtc.so.12. Error: libnvrtc.so.12: cannot open shared object file: No such file or directory Could not load library libnvrtc.so.12. Error: libnvrtc.so.12: cannot open shared object file: No such file or directory Could not load library libnvrtc.so.12. Error: libnvrtc.so.12: cannot open shared object file: No such file or directory /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/autograd/graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at ../aten/src/ATen/native/cudnn/MHA.cpp:674.) return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass steps: 0%| | 1/48000 [00:12<163:06:27, 12.23s/it, avr_loss=0.384]Could not load library libnvrtc.so.12. Error: libnvrtc.so.12: cannot open shared object file: No such file or directory Could not load library libnvrtc.so.12. Error: libnvrtc.so.12: cannot open shared object file: No such file or directory Could not load library libnvrtc.so.12. Error: libnvrtc.so.12: cannot open shared object file: No such file or directory Could not load library libnvrtc.so.12. Error: libnvrtc.so.12: cannot open shared steps: 0%|

SECourses

if working yes you can ignore. also i updated torch 2.5.1 it should fix this

Steve

Ty!

Colab

If I use batch size 1, and have 63 image dataset, how many epochs or steps should I aim for? The configuration file suggests 200 epochs which would be over 12000 steps. I thought 1000 steps was said to be optimal?

SECourses

i would aim 100-150 , depending on how much you can wait. no such thing as 1000 steps optimal. you can see i did way more with my 256 images dataset : https://youtu.be/FvpWy1x5etM

SECourses

sadly i dont know. last time i tested it was working. i need to test again later. could be that pod broken or kohya broken. did you try on massed compute?

Tycjan Gniew

Hi, I'm training FLUX LoRA on massed compute with 1x RTX A6000 [ALT Config] on 15 images with 2000 steps. Best_Configs/Rank_1_29500MB_8_85_Second_IT.json is my configuration and so far everything is going fine, except that checkpoints saved every 25 epochs is 2.5 GB. As far as i know the whole point of LoRA is that it should be realy small file. What bam I doing wrong?

SECourses

Nothing wrong. We save as float thus size is 2x. you can save as fp16 to half the size. you can also later save as fp8 via kohya to half the size again.

kartonpat

Hello and thank you for your work :) When i launch the GUI I get this : Starting the GUI... this might take some time... Traceback (most recent call last): File "C:\Users\Windows\Documents\kohya flux\kohya_ss\kohya_gui.py", line 6, in import gradio as gr ModuleNotFoundError: No module named 'gradio' (venv) C:\Users\Windows\Documents\kohya flux\kohya_ss> I checked and updated gradio but I still get this message. Any idea ? THank you

kartonpat

Hello and thank you for all this nice work! when trying to download the models from your file i get this : clip_l.safetensors: 100%|███████████████████████████████████████████████████████████| 246M/246M [00:06<00:00, 38.0MB/s] Could not set the permissions on the file 'C:\kohya flux 2\.cache\huggingface\download\clip_l.safetensors.660c6f5b1abae9dc498ac2d21e1347d2abdb0cf6c0c0c8576cd796491d9a6cdd.incomplete'. Error: [Errno 13] Permission denied: 'C:\\tmp_cb40941d-37f7-4e01-a619-ee5e315fbe64'.|█▌ | 273M/9.79G [00:06<03:42, 42.8MB/s] Continuing without setting permissions. ae.safetensors: 100%|███████████████████████████████████████████████████████████████| 335M/335M [00:08<00:00, 38.8MB/s] Could not set the permissions on the file 'C:\kohya flux 2\.cache\huggingface\download\ae.safetensors.afc8e28272cd15db3919bacdb6918ce9c1ed22e96cb12c4d5ed0fba823529e38.incomplete'. Error: [Errno 13] Permission denied: 'C:\\tmp_32e18f27-7dfe-498f-a69c-7722f8ece12f'. 4%|█▉ | 357M/9.79G [00:08<03:42, 42.4MB/s] Continuing without setting permissions. Is it normal ?

SECourses

it is definitely not normal. something on your system is blocking hugging face downloader. probably your antivirus . try again and it should resume but you should allow python code to run

SECourses

install error. also i would never install there c user windows . please follow this video to install your requirements properly first : https://youtu.be/DrhUHnYfwC0 install everything directly into C drive

kartonpat

I did, thank you !

kartonpat

Edited: I don't have an anti virus set on this computer. I have a GTX 4080 16go and used the ranked 8 config. Is it the right one ? Because it changed the setting to float instead of bf16 which seams better. And the size is now 768x768. What do you think ?

SECourses

when i last tested fp16 were not learning. but if results are good all good

kartonpat

Thank you again. This is so annoying when it does not work… I managed to train the Lora. But when I use flux1.dev with forge and Lora. I get errors in forge. « AssertionError you do not have clip state dict! » and sometimes a message about missing object I don’t remember. I am so sorry to bother you. Lora training is extremely important for my work. Thank you

SECourses

well with my config trained loras, not changed except like number of epochs, all of them works with swarmui. so impossible to know without further debug. if you upgrade gold member i can setup a training on your computer quickly if you are not doing anything NSFW

Bayram TATAR

Hello Mr. Furkan... I am using i5 12th generation processor and RTX 4060 8GB vram and 16GB ram available on the pc (cpu). I have examined the configurations here. But I couldn't fully understand. I don't know much about this. Can I do this training with this computer. and which configuration file would you recommend me to use. thank you. Note: By the way, I am in Samsun, Turkey. you can answer in Turkish.

SECourses

Bayram bey selamlar. Ekran kartınız VRAM olarak zayıf ama daha büyük problem sadece 16 GB RAM olması. 2x32 GB RAM alırsanız, LoRA yerine direkt fine tuning yaparak harika sonuçlar elde edebilirsiniz. LoRA da gene fena çalışmaz : Rank_12_7500MB_7_68_Second_IT.json

Bayram TATAR

teşekkür ederim. sanırım bu sistemle yapamıyacağız. birde birkaç kez denememe rağmen şöyle bir hata alıyorum. bunun çözümü var mıdır acaba. tam anlayamıyorum... kohya derambooth da : enable full bf16 training. Traceback (most recent call last): File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\flux_train.py", line 998, in train(args) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\sd-scripts\flux_train.py", line 453, in train flux = accelerator.prepare(flux, device_placement=[not is_swapping_blocks]) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1311, in prepare result = tuple( File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1312, in self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1188, in _prepare_one return self.prepare_model(obj, device_placement=device_placement) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 1435, in prepare_model model = model.to(self.device) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1340, in to return self._apply(convert) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply module._apply(fn) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply module._apply(fn) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply module._apply(fn) [Previous line repeated 1 more time] File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 927, in _apply param_applied = fn(param) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1326, in convert return t.to( torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 54.00 MiB. GPU 0 has a total capacity of 8.00 GiB of which 0 bytes is free. Of the allocated memory 14.59 GiB is allocated by PyTorch, and 17.75 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) Traceback (most recent call last): File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in sys.exit(main()) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "D:\Kohya_GUI_Flux_Installer_v42\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['D:\\Kohya_GUI_Flux_Installer_v42\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/Kohya_GUI_Flux_Installer_v42/kohya_ss/sd-scripts/flux_train.py', '--config_file', 'D:/LOraY\\model/config_dreambooth-20241109-125149.toml']' returned non-zero exit status 1. 12:56:29-151176 INFO Training has ended.

SECourses

evet yetersiz VRAM hatası ayrıca sizde RAM'de yok

Quốc Tuấn Dương

About .txt caption files, I noticed that you just put 1 .txt file in your photo folder. When I used Joy cation, there were 15 caption files for my 15 photos. Should I put 15 caption files to my photo folder or just 1 file?

SECourses

it was for 1 trainer i didnt use it. the kohya works in 2 ways. either reads folder name as caption or you provide image_name.txt for every image and write caption in that

baz64565 .

I've tried the rank12 8GB config, but i'm still getting out of memory, only 32MB though, I've already closed everything running that is using GPU except for default windows processes : return t.to( torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacity of 8.00 GiB of which 0 bytes is free. Of the allocated memory 7.31 GiB is allocated by PyTorch, and 14.35 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

SECourses

i really recommend to do fine tuning instead of lora. it will be way better. also a huge speed up is about to arrive to the fine tuning. i just messaged kohya to merge branch. once updated it will use only 6900 MB VRAM. but current configs and version also works - just slower

baz64565 .

I see, how much time difference are we talking about between lora and finetune? the lora usually takes about 2h40 on 8GB (fluxgym), not sure how much the rank12 on kohya_ss normally took with 15 images.

SECourses

if you are willing to lose quality you can even done it faster. you want quality or speed that matters. my configs targets best quality. and none of the lora will ever reach of fine tuning quality.

baz64565 .

Ok I will try out the finetune when it gets updated to be a bit faster.

Magica Barłogi

Hi. I have a problem, probably quite childish. I don't know how to resume training from the last checkpoint. I have a 125 epoch chatpoint, I have a .safetensors file. Could someone help me? How can I load it and resume training?

Yash Rami

I want to train a shirt's LoRA, so will it be called as style or character LoRA? Also how many training images will be required, with or without captions??

baz64565 .

I've started the 7200MB training config, it's going to take +- 12hours. Which parameters can I change to speed up and sacrifice some quality? Is it network dim and network alpha settings? Or is there a better setting to reduce to get it atleast around 6 hours, maybe max 8 hours.

baz64565 .

In the basic options you have a little optional field called "Network weights" Path to an existing LoRA network weights to resume training from -> give the path to your last trained epoch there

baz64565 .

I found out why I OOM'd, I was using the FP16 t5xxl and it was crashing when it had to be loaded into memory. using the FP8 now.

SECourses

dont use captions. if you want shirt it will be a shirt lora. i would try to gather like 30 40 good poses distances angles

SECourses

what is your GPU? reducing resolution will bring most speed up during LoRA training - when you reduce resolution you can also reduce block swap count

baz64565 .

its a 3060ti 8GB , i'm using 512x512 pictures, but i notice the config settings is at 896,896 and the bucket_max is at 2048, do those causing slower times even though i'm using only 512 in my training set? what would be an idea swap count instead?

SECourses

yes do this. change resolution in config to 512x512, copy images into a new folder and use that folder. bucket size not important - try swap count 1 by 1 like decrease 1 try decrease again try and such

SECourses

thanks accurate answer. also it will show as started from 0 but it will be actually starting from 125. so make new epoch count accordingly

baz64565 .

changing to 512,512 and block from 35 -> 30 , made time drop from 12h to 4h ;) Atleast that can finish before I have to go to bed, i don't like long run times, because I don't like the possibility of my PC catching on fire while i'm sleeping or away from home. I know you mostly concentrate on quality, which is great, but a small note in tutorial for those that want to tweak the balance for speed/quality a little, might help a lot of others too, like you did for me just now. I think in the beginning I also decreased network dim and alpha from 128 to 64, because I used to get OOMs and I thought it was caused by the high network dim. Could that be the reason i'm not at high GPU in use , but instead 5.7/8GB dedicated gpu memory used?

baz64565 .

Ok , i quit the training and reset dimensions to 128, running stable now at 7gb/8gb used, shared mem at 12.3GB , swap at 30, resolution 512x512, 16 images, 1 repeat, 200 epoch. 4h10 on 3060ti 8GB

SECourses

4 hours is awesome good job there

baz64565 .

I just finished, it works, but I had to use previous epochs instead of final one, it overtrained fast and the character becamse "noisy", i trained 3200 steps with 16 images and the last satisfying useable epoch was around 2100 steps. So I could have stopped around 3 hours even.

SECourses

yep. hard to know exactly which checkpoint will be best

Sonivas Sx

got this error after last update "flux_train_network.py: error: unrecognized arguments: --blocks_to_swap 26 Traceback (most recent call last): File "C:\Python3_10_11\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python3_10_11\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Kohya\Kohya_GUI_Flux_Installer_27\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in File "C:\Kohya\Kohya_GUI_Flux_Installer_27\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "C:\Kohya\Kohya_GUI_Flux_Installer_27\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "C:\Kohya\Kohya_GUI_Flux_Installer_27\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\\Kohya\\Kohya_GUI_Flux_Installer_27\\kohya_ss\\venv\\Scripts\\python.exe', 'C:/Kohya/Kohya_GUI_Flux_Installer_27/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'C:\\Users\\S-Pc\\Desktop\\Training Outputs\\model/config_lora-20241118-021525.toml', '--blocks_to_swap', '26']' returned non-zero exit status 2."

SECourses

let me update configs bmaltais probably fixed his error

SECourses

ok i just tested and your kohya is very old. please make a fresh install. use latest zip file. use Windows_Install_Step_1.bat and then Windows_Install_Torch_2_5_Dev_Huge_Speed_Up.bat and then Windows_Start_Kohya_SS.bat

Arder

I have 4070ti super with 16gb. When I try use new config 16GB_GPU_Quality_Tier_3_14700MB_9.2_Second_IT I recieve this error: enable full bf16 training. Traceback (most recent call last): File "D:\AI\kohya\kohya_ss\sd-scripts\flux_train_network.py", line 574, in trainer.train(args) File "D:\AI\kohya\kohya_ss\sd-scripts\train_network.py", line 639, in train unet = self.prepare_unet_with_accelerator(args, accelerator, unet) # accelerator does some magic here File "D:\AI\kohya\kohya_ss\sd-scripts\flux_train_network.py", line 545, in prepare_unet_with_accelerator accelerator.unwrap_model(flux).prepare_block_swap_before_forward() File "D:\AI\kohya\kohya_ss\sd-scripts\library\flux_models.py", line 1006, in prepare_block_swap_before_forward self.offloader_single.prepare_block_devices_before_forward(self.single_blocks) File "D:\AI\kohya\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 210, in prepare_block_devices_before_forward weighs_to_device(b, "cpu") # make sure weights are on cpu File "D:\AI\kohya\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 91, in weighs_to_device module.weight.data = module.weight.data.to(device, non_blocking=True) RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Traceback (most recent call last): File "C:\Users\Arder\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\Arder\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "D:\AI\kohya\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in sys.exit(main()) File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "D:\AI\kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['D:\\AI\\kohya\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/AI/kohya/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'D:/AI/SwarmUI/SwarmUI/Models/Lora/config_lora-20241118-023205.toml', '--blocks_to_swap', '23']' returned non-zero exit status 1. switch to fp8 instead bf16 work fine and fast 6.65it. Does it mean bf165 doesnt work with 16gb or I missing something in configs that could make it work?

SECourses

how much RAM you have? it could be reason. you might be needed 64 GB RAM

Sonivas Sx

so should i download kohya again to get the new configs or just update?

SECourses

you should do a fresh install. after that use the new attached Fix_Kohya_GUI_Error_Temporary.bat file it will reverse to older working commit. it is error of bmaltais - kohya gui maker

Sonivas Sx

since its been a while i do install step 1,5,6 should i do Triton as well ? heard some news lately about speeding things up

SECourses

not needed. so Windows_Install_Step_1.bat - and then option 1 and 5. then use Windows_Install_Torch_2_5_Dev_Huge_Speed_Up.bat and then use our starter to start. also dont forget temporary fix for a while now

Sonivas Sx

ok doing a fresh install and after step 1 i get this and cant move to 5 Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "C:\Kohya\Kohya_GUI_Flux_Installer_v44\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 4, in File "C:\Kohya\Kohya_GUI_Flux_Installer_v44\kohya_ss\venv\Lib\site-packages\accelerate\commands\accelerate_cli.py", line 19, in from accelerate.commands.estimate import estimate_command_parser File "C:\Kohya\Kohya_GUI_Flux_Installer_v44\kohya_ss\venv\Lib\site-packages\accelerate\commands\estimate.py", line 34, in import timm File "C:\Kohya\Kohya_GUI_Flux_Installer_v44\kohya_ss\venv\Lib\site-packages\timm\__init__.py", line 2, in from .models import create_model, list_models, is_model, list_modules, model_entrypoint, \ File "C:\Kohya\Kohya_GUI_Flux_Installer_v44\kohya_ss\venv\Lib\site-packages\timm\models\__init__.py", line 28, in from .maxxvit import * File "C:\Kohya\Kohya_GUI_Flux_Installer_v44\kohya_ss\venv\Lib\site-packages\timm\models\maxxvit.py", line 225, in @dataclass ^^^^^^^^^ File "C:\Python312\Lib\dataclasses.py", line 1275, in dataclass return wrap(cls) ^^^^^^^^^ File "C:\Python312\Lib\dataclasses.py", line 1265, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python312\Lib\dataclasses.py", line 994, in _process_class cls_fields.append(_get_field(cls, name, type, kw_only)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python312\Lib\dataclasses.py", line 852, in _get_field raise ValueError(f'mutable default {type(f.default)} for field ' ValueError: mutable default for field conv_cfg is not allowed: use default_factory [11/18/24 13:04:08] ERROR Error occurred while running command: accelerate config default setup_common.py:673 ERROR Error: Command 'accelerate config default' returned non-zero exit status 1.

Arder

Thanks! Yes I have 32gb Ram, so I will stick with fp8 for the moment, how much quality drop between fp8 and bf16 it shouldnt be big with 1024 resolution?

SECourses

yes it is minimal difference. 1024 is best resolution

Sonivas Sx

now i get the same error as Arder. i have 32Gb ram and i had no problem in the past with a 4080, how to change to fp8 is it the option from number 5 in intallation or using the fp8 model of flux?

Arder

In parameters -> Advanced remove checkbox from "Full bf16 training (experimental)" and put on "fp8 base"

SECourses

so using that fp8 technique with which workflow you made it work on 32 gb ram?

SECourses

i think i will add fp8 model download and fp8 option checkmark to the fp8 configs

baz64565 .

It seems I was too early cheering. The lora works, but very limited in use, only close ups are amazing, once the character is a medium distance shot the face looks like it gets too much noise, the same thing happens when combining with other loras, at first I thought it was the epoch that overtraind, but I went back from epoch 200 all the way to 40 and they have it all, I will try again with maybe network dim 64 and alpha 32, I read somewhere someone else had a similar issue and it was because of 128 rank

Arder

Yes, I only removed "Full bf16 training (experimental)" from 6GB_GPU_Quality_Tier_3_14700MB_9.2_Second_IT and it works on 32gb ram

SECourses

wow that is new for me. thanks a lot @Arder

SECourses

are you inpainting the face? you really should do with distant and mid shots as i do in tutorial

yijie fang

Hello and thank you for your work :) . I’m trying to train a LoRA model for a down jacket, but it struggles with details like buttons and zippers. Do you have any good solutions for this?

SECourses

yes bigger resolution training and better dataset

yijie fang

Thanks! I will try it :) . I previously tried adding close-up images of the clothing, such as the lining, to the dataset, and it did improve the generated results. Later, I attempted to assign different triggers to each part, including the button on the left and the buttonhole on the right (asymmetrical). When I only added close-ups of one side, I found that LoRA could learn it, but it would turn both the left and right buttons into the appearance of the single close-up. However, when I added detailed close-ups of both sides, I found that LoRA couldn’t distinguish between the left and right sides. May I kindly ask if you have any advice or experience with this approach to incorporating close-up data?

Sonivas Sx

Thanks to both of you. I really just went with tier 4 of 16gb but based on what you say maybe i can go for tier 2 and change to fp8 base in parameters. Also is this https://github.com/woct0rdho/triton-windows integrated in kohya or is the option of Triton in kohya different? It took 2hr 10m for 24 images on 4080 and 32gb with Dev flux / 1 repeat/ 1600 steps for a LORA which is crazy . what i dont remember though is which option i chose when installing kohya in number 5 either bf16 or fp8. i will try with bigger dataset like the one you did Dr. F and maybe with the fp8 model as well !!!

SECourses

i think triton not used during training. we are already getting linux speed on windows with torch 2.5.1

baz64565 .

I found it hard to explain what I was seeing, but i found this on reddit and its exactly whats happening to me when using the loras from kohya training. A sort of patchwork pattern overlayed on the image creating noise and unneccesary sharpness. It seems AI-toolkit fixed it, could it still be present in kohya code base? Others say its a flux problem when going above 1024x1024 and there's not really a solution and that it gets worse when combining multiple loras https://www.reddit.com/r/StableDiffusion/comments/1es91bu/comment/li49tvp/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

SECourses

great let us know. also triton not using training as far as i know

SECourses

i wouldnt add such close ups. it may confuse as you noticed. instead try like 1536px training

GH

Wait what? I thought the max image resolution we could use to train was 1024px? Wouldn't 1536px training images just get resized to 1024px by kohya automatically?

Markus Lienert

Wow, many thanks for the speedy updates! I create sample promts during training with a 24GB RTX 3090. 24GB_GPU_Quality_Tier_1_22950MB_14.1_Second_IT_Trains_T5_and_T5_Attention unfortunately runs out of memory. With blocks_to_swap 33 it worked. Best regards!

Brett Kelly

8GB GPU Ram, using your Tier4 and Tier5 scripts and I am getting out of memory. All startup stuff disabled and I have full free VRAM before starting, any ideas?

SECourses

How much RAM you have in your computer?

SECourses

if you set max resolution to 1536 they wont get downscaled to 1024

baz64565 .

make sure you load a T5xxl that is FP8, the configs for 8GB wil OOM if you load the default T5xxl which is fp16

Brett Kelly

I have 16GB, I tried T5XXL FP8 still out of memory. Tried to increase the virtual ram but made no difference unfortunately.

GH

If I'm training with 1536x1536px on an RTX 3090, how should I modify the settings of '24GB_GPU_22900MB_7.3_second_it_Tier_1', as currently switching from 1024px to 1536px goes from 8s/it to 120s/it. I've tried increasing the block swap from 9 to 16. I can try increase it some more, but is there any other setting I could change to bring this s/it down without impacting quality for 1536px training? Thanks Dr

Brett Kelly

SECourses, any ideas why it not run on 8GB as you say it can?

SECourses

You have far too low RAM. you need to upgrade. get 2x 32gb and you will run very good

SECourses

you have to increase block swap since it will use far more VRAM. increase 1 by 1 and test at least 10 steps and see which one work fastest

SECourses

hi it is removed since not necessary anymore. I am adding removing bat files as being necessary :) now we have Windows_Kohya_Update.bat

Brett Kelly

I have virtual ram, surely that should be sufficient?

Brett Kelly

Why is the error cuda out of memory? Also I can run on onetrainer fine, no problem so should not be ram issue. Did you even test configs on these VRAM? Even the 6GB VRAM config says out of cuda memory...

Leonard Bruno

Professor, in the kohya parameters section I am missing Flux as an output Lora type. I noticed you use Flux1 in your training video. How do I get Flux1 to show up here as a dropdown option?

baz64565 .

I found another post and I will retry the training and see if it solved the issue. "train_double_block_indices": "0,4-18" , it removes some of the blocks that are apparently the culprit for the pattern appearing. So blocks 1 to 3 are removed, they supposedly affect composition a little but not style and likeness.

kartonpat

Hello ! Please help :) Everything was working quite fine and I wanted to install the new version. To be sure I was not messing with my working version I created a new file. I clicked on Windows_Install_Step_1.bat ... I then clicked on going back to cuda 2.5.0 and now nothing will work no matter what. Here is what I get. I'm sorry. Thank you Traceback (most recent call last): File "C:\kohya_flux_3\kohya_ss\sd-scripts\flux_train_network.py", line 574, in trainer.train(args) File "C:\kohya_flux_3\kohya_ss\sd-scripts\train_network.py", line 639, in train unet = self.prepare_unet_with_accelerator(args, accelerator, unet) # accelerator does some magic here File "C:\kohya_flux_3\kohya_ss\sd-scripts\flux_train_network.py", line 545, in prepare_unet_with_accelerator accelerator.unwrap_model(flux).prepare_block_swap_before_forward() File "C:\kohya_flux_3\kohya_ss\sd-scripts\library\flux_models.py", line 1006, in prepare_block_swap_before_forward self.offloader_single.prepare_block_devices_before_forward(self.single_blocks) File "C:\kohya_flux_3\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 210, in prepare_block_devices_before_forward weighs_to_device(b, "cpu") # make sure weights are on cpu File "C:\kohya_flux_3\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 91, in weighs_to_device module.weight.data = module.weight.data.to(device, non_blocking=True) RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Traceback (most recent call last): File "C:\Users\Windows\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\Windows\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\kohya_flux_3\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in sys.exit(main()) File "C:\kohya_flux_3\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "C:\kohya_flux_3\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "C:\kohya_flux_3\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\\kohya_flux_3\\kohya_ss\\venv\\Scripts\\python.exe', 'C:/kohya_flux_3/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'C:/Users/Windows/Pictures/voile satin - Copie\\model/config_lora-20241122-214627.toml']' returned non-zero exit status 1. 21:48:00-227345 INFO Training has ended.

SECourses

so easy. when you select the downloaded 23.8 gb flux model in the model selection path it will appear

SECourses

hi this is out of VRAM error. your config could be corrupted. can you tell me GPU and get a fresh config and try again please

SECourses

if you have 64 GB RAM 100%. if you have lesser i can't say for sure. system ram 64 gb works

Mehmet Atakan Çavuşlu

Hello Furkan, my installation gets stuck when using your .zip file. It stops exactly at wheel creation for fairscale (i dont know what it is) but after this line it stops continuing even if I wait a lot: Here is the last line in cmd window, can you help?: INFO Building wheels for collected packages: fairscale, schedulefree INFO Building wheel for fairscale (pyproject.toml): started INFO Building wheel for fairscale (pyproject.toml): finished with status 'done' INFO Created wheel for fairscale: filename=fairscale-0.4.13-py3-none-any.whl size=332117 sha256=39ab905f6375b7e194e6dd91f3cea3a75c09351518f6c60f84b0abf4f59ad78 a

4401 4401

just wondering, I've just made a new lora and it's over 3gb, it's a fairly simple character model, similar ones I see on civitai are usually a matter of megabytes rather than gigabytes, anyway to make these smaller?

Shepard2k

Greetings, I just tried your Config "48GB_GPUz_4x_GPU_Quality_Tier_3_29750MB_5.7_Second_IT" from the your current Installer v46 and it didn't work, error messages before training started. Tried on Runpod with 4xA40. Maybe when you have the time you can check if this is a general problem or I f*cked up something. I'm currently back to your "4x_GPU_Rank_1_SLOW_Better_Quality" from v42 and it seems to work. Nevertheless thanks again for providing all your Auto-Installers for Lazy-Noobs like me.

SECourses

thanks a lot. what was the error you got with new config?

Justas

I've tried to save my custom configs on top of yours (even save as new one) but when I loaded them again, some fields were still set to your defaults (i.e. Clip-l or T5 were leading to Ubuntu/downloads/... or so). Save button is not overwriting them, you might need update kohya a bit :)

SECourses

well i refresh gradio interface before loading configs or it get messed up :D

Justas

I've loaded them on a freshly restarted PC, so I assume it's something about saving rather than loading, but it's a minor issue. Question regarding the Lora training: Say, I've got a prepared image dataset of a person (30-40 random pics of him). What number of steps should I be aiming to do locally? For the last couple of months, I've been training Loras on Civitai with the following settings: 7 repeats, 15 epochs, 3,4-4,2k steps, LR/Alpha 32 and the results were quite good. What would be your recommendations please?

SECourses

well with my config for 40 images i would recommend 200 epochs. so 8000 steps. but make sure save checkpoints and compare. probably around 150 epochs will be best but it is subjective - repeating is always 1 with flux since we dont use reg images

Gavin Goodman

I'm halfway through but cannot get a successful training to run. I have made sure my training dataset of images is in the correct folder many times ? Start training LoRA Flux1 ... 14:13:11-621828 INFO Validating lr scheduler arguments... 14:13:11-622537 INFO Validating optimizer arguments... 14:13:11-623546 INFO Validating lora type is Flux1 if flux1 checkbox is checked... 14:13:11-623546 INFO Validating D:/Sofia/output existence and writability... SUCCESS 14:13:11-624545 INFO Validating D:/Khoya/Kohya_GUI_Flux_Installer_v46/flux1-dev.safetensors existence... SUCCESS 14:13:11-624545 INFO Validating D:/Sofia/trained/test existence... SUCCESS 14:13:11-625544 INFO Regularization factor: 1 14:13:11-625544 INFO Train batch size: 1 14:13:11-626545 INFO Gradient accumulation steps: 1 14:13:11-626545 INFO Epoch: 200 14:13:11-627546 INFO max_train_steps (0 / 1 / 1 * 200 * 1) = 0 14:13:11-628545 INFO stop_text_encoder_training = 0 14:13:11-629545 INFO lr_warmup_steps = 0 14:13:11-630545 INFO Saving training config to D:/Sofia/output\Rank_2_27360MB_Fast_20241203-141311.json... 14:13:11-631546 INFO Executing command: D:\Khoya\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no --dynamo_mode default --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 D:/Khoya/Kohya_GUI_Flux_Installer_v46/kohya_ss/sd-scripts/flux_train_network.py --config_file D:/Sofia/output/config_lora-20241203-141311.toml D:\Khoya\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( D:\Khoya\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( 2024-12-03 14:13:18 INFO Loading settings from train_util.py:4519 D:/Sofia/output/config_lora-20241203-141311.toml... INFO D:/Sofia/output/config_lora-20241203-141311 train_util.py:4538 INFO highvram is enabled / highvramが有効です train_util.py:4190 2024-12-03 14:13:18 INFO Checking the state dict: Diffusers or BFL, dev or schnell flux_utils.py:43 INFO t5xxl_max_token_length: 512 flux_train_network.py:141 D:\Khoya\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884 warnings.warn( You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 2024-12-03 14:13:19 INFO Using DreamBooth method. train_network.py:325 INFO prepare images. train_util.py:1971 INFO 0 train images with repeating. train_util.py:2012 INFO 0 reg images. train_util.py:2015 WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:2020 INFO [Dataset 0] config_util.py:567 batch_size: 1 resolution: (1024, 1024) enable_bucket: False network_multiplier: 1.0 INFO [Dataset 0] config_util.py:573 INFO loading image sizes. train_util.py:923 0it [00:00, ?it/s] INFO prepare dataset train_util.py:948 ERROR No data found. Please verify arguments (train_data_dir must be the train_network.py:366 parent of folders with images) / 画像がありません。引数指定を確認してください(train_data_dirには画像が あるフォルダではなく、画像があるフォルダの親フォルダを指定する必要があ ります) 14:13:20-905473 INFO Training has ended.

SECourses

hi as i replied it is image dataset folder naming error : 0 train images with repeating.

Michael Liu

hi prof, im using RTX 4060 TI 16gigs vram and when i load your config 16GB_GPU_Quality_Tier_3_14700MB_9.2_Second_IT. the train ended right after i start the training WARNING constant_with_warmup will be good / train_util.py:4873 スケジューラはconstant_with_warmupが良いかもしれません enable full bf16 training. Traceback (most recent call last): File "C:\Kohya\kohya_ss\sd-scripts\flux_train_network.py", line 583, in trainer.train(args) File "C:\Kohya\kohya_ss\sd-scripts\train_network.py", line 664, in train unet = self.prepare_unet_with_accelerator(args, accelerator, unet) # accelerator does some magic here File "C:\Kohya\kohya_ss\sd-scripts\flux_train_network.py", line 554, in prepare_unet_with_accelerator accelerator.unwrap_model(flux).prepare_block_swap_before_forward() File "C:\Kohya\kohya_ss\sd-scripts\library\flux_models.py", line 1006, in prepare_block_swap_before_forward self.offloader_double.prepare_block_devices_before_forward(self.double_blocks) File "C:\Kohya\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 210, in prepare_block_devices_before_forward weighs_to_device(b, "cpu") # make sure weights are on cpu File "C:\Kohya\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 91, in weighs_to_device module.weight.data = module.weight.data.to(device, non_blocking=True) RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Traceback (most recent call last): File "C:\Users\MTP023\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\MTP023\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Kohya\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in sys.exit(main()) File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "C:\Kohya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\\Kohya\\kohya_ss\\venv\\Scripts\\python.exe', 'C:/Kohya/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'C:\\fluxtraining/train_imgs\\model/config_lora-20241205-101408.toml']' returned non-zero exit status 1. 10:15:30-067542 INFO Training has ended.

SECourses

Hi if you have 32 GB RAM please upgrade to 64 GB and it will be 100% fixed

SECourses

yes we have newer configs if you have 64 GB RAM for 8 GB and thus it can be

秀煜玩遊戲 Show YoU Gaming

I tried to train a size 768,512 img in my 3060ti VRAM and 68G RAM device it shows like this. enable full bf16 training. running training / 学習開始 num examples / サンプル数: 70 num batches per epoch / 1epochのバッチ数: 70 num epochs / epoch数: 200 batch size per device / バッチサイズ: 1 gradient accumulation steps / 勾配を合計するステップ数 = 1 total optimization steps / 学習ステップ数: 14000 steps: 0%| | 0/14000 [00:00 train(args) File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\sd-scripts\flux_train.py", line 682, in train accelerator.backward(loss) File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py", line 2159, in backward loss.backward(**kwargs) File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\torch\_tensor.py", line 581, in backward torch.autograd.backward( File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\torch\autograd\__init__.py", line 347, in backward _engine_run_backward( File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py", line 825, in _engine_run_backward return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py", line 1125, in unpack_hook frame.recompute_fn(*args) File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py", line 1519, in recompute_fn fn(*args, **kwargs) File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\sd-scripts\library\flux_models.py", line 833, in _forward attn = attention(q, k, v, pe=pe, attn_mask=attn_mask) File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\sd-scripts\library\flux_models.py", line 452, in attention x = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 96.00 MiB. GPU 0 has a total capacity of 8.00 GiB of which 0 bytes is free. Of the allocated memory 9.52 GiB is allocated by PyTorch, and 818.46 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) steps: 0%| | 0/14000 [00:48 sys.exit(main()) File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "D:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['D:\\Kohya_GUI_Flux_Installer_v46\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/Kohya_GUI_Flux_Installer_v46/kohya_ss/sd-scripts/flux_train.py', '--config_file', 'D:\\train_imgs\\model/config_dreambooth-20241205-234453.toml']' returned non-zero exit status 1. 23:54:37-269084 INFO Training has ended. 23:58:29-068957 INFO Loading config...

SECourses

hi can you tell me which config file you used exactly? you shouldnt modify config files configuration. please let me know config name. with 68 gb ram it should work but are you usre you have 68 gb RAM ? i doubt that you have it . can it be 64 ?

Justin

DUDDDDDDDDDDE! Thank you so much. I could not figure out this out. No matter what I did, I was getting this error and I was not able to solve the issue. VRAM memory error!

Marc Mercuri

During install, I'm getting the following error (note - that this is running locally on a Windows Machine . The machine has multiple versions of python installed, but is set to 3.11 in the venv. I changed this manually as even though environment variables had 311 listed first in the path, it kept pinning to an early 3.10 version which was outside of the range in the script) Any thoughts on how to remediate the issue? [12/09/24 04:51:15] INFO Kohya_ss GUI version: v24.2.0 setup_common.py:372 INFO Python version is 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 setup_common.py:28 bit (AMD64)] INFO Submodule initialized and updated. setup_common.py:53 INFO Installing/Validating requirements from setup_common.py:161 requirements_pytorch_windows.txt... [12/09/24 04:51:16] INFO Looking in indexes: https://pypi.org/simple, setup_common.py:183 https://pypi.ngc.nvidia.com, https://download.pytorch.org/whl/cu124 INFO Obtaining file:///A:/Kohya_GUI_Flux_Installer_v46/kohya_ss/sd-scripts setup_common.py:183 (from -r requirements.txt (line 37)) INFO Preparing metadata (setup.py): started setup_common.py:183 [12/09/24 04:51:17] INFO Preparing metadata (setup.py): finished with status 'done' setup_common.py:183 [12/09/24 04:51:38] INFO Installing collected packages: library setup_common.py:183 INFO Attempting uninstall: library setup_common.py:183 INFO Found existing installation: library 0.0.0 setup_common.py:183 INFO Uninstalling library-0.0.0: setup_common.py:183 [12/09/24 04:51:41] INFO Successfully uninstalled library-0.0.0 setup_common.py:183 INFO Running setup.py develop for library setup_common.py:183 [12/09/24 04:51:42] INFO Successfully installed library-0.0.0 setup_common.py:183 Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "A:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 4, in File "A:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\Lib\site-packages\accelerate\commands\accelerate_cli.py", line 19, in from accelerate.commands.estimate import estimate_command_parser File "A:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\Lib\site-packages\accelerate\commands\estimate.py", line 34, in import timm File "A:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\Lib\site-packages\timm\__init__.py", line 2, in from .models import create_model, list_models, is_model, list_modules, model_entrypoint, \ File "A:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\Lib\site-packages\timm\models\__init__.py", line 28, in from .maxxvit import * File "A:\Kohya_GUI_Flux_Installer_v46\kohya_ss\venv\Lib\site-packages\timm\models\maxxvit.py", line 225, in @dataclass ^^^^^^^^^ File "C:\Users\mmerc\AppData\Local\Programs\Python\Python311\Lib\dataclasses.py", line 1221, in dataclass return wrap(cls) ^^^^^^^^^ File "C:\Users\mmerc\AppData\Local\Programs\Python\Python311\Lib\dataclasses.py", line 1211, in wrap return _process_class(cls, init, repr, eq, order, unsafe_hash, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\mmerc\AppData\Local\Programs\Python\Python311\Lib\dataclasses.py", line 959, in _process_class cls_fields.append(_get_field(cls, name, type, kw_only)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\mmerc\AppData\Local\Programs\Python\Python311\Lib\dataclasses.py", line 816, in _get_field raise ValueError(f'mutable default {type(f.default)} for field ' ValueError: mutable default for field conv_cfg is not allowed: use default_factory [12/09/24 04:52:05] ERROR Error occurred while running command: accelerate config default setup_common.py:674 ERROR Error: Command 'accelerate config default' returned non-zero exit setup_common.py:675 status 1.

SECourses

you are still using python 3.11. you need to use python 3.10 : "C:\Users\mmerc\AppData\Local\Programs\Python\Python311\Lib\dataclasses.py", line 959, in _process_class

Александр Щербинин

Good afternoon. Tell me, please, does torch 2.5.1 affect the quality of generation? It feels as if after the installation of the "Lora" they began to work worse.

SECourses

use 2.5.0 it works best on windows

Александр Щербинин

We would like to hear your opinion. There is a service https://www.basedlabs.ai/ that trains Lora in about 7-8 minutes. 2000 steps / Number of photos: 30-50, 1024*1024. The results are quite good, and we would like to achieve the same level or better, but locally or through Runpod. The task is to train a large number of different people in a short period of time. For some reason, the result was worse with your configs ((

Александр Щербинин

It's not that you have bad configs. I see that everything works well and the examples are impressive. I thought maybe you already have a ready-made solution or you know how such services work))

SECourses

yes i know how they work. they use expensive GPU like H100, very high batch size, higher learning rate and possibly training only certain layers instead of full LoRA. all these reduces quality but brings speed. they could be even using multiple GPUs since you almost get linear speed increase for LoRA

인폴 레

When I chose "1.Install kohya_ss GUI" Menu in "Kohya_ss setup menu" and enter the CMD window closes. I can't install anymore. what's problem?

Александр Щербинин

I found the settings I needed thanks to your configs. Not in 5-7 minutes, but also quickly. It remains to make improvements. Thank you.

SECourses

open a cmd window in that folder, type the bat file name fully and start that way. when you do that way it wont auto close and you will see the error reason

努力生活的超子

Hello, I want to use Flux to train a style Lora model. I now have 100 different character pictures (white background) of the same style and 100 scene pictures. I will use a 4090 graphics card for training. How do you think I should choose the data? Looking forward to your reply~ ❤️ ❤️

SECourses

well it is purely experience based. but for style training only style should repeat in training images no other subjects or objects. also style should be consistent. these are 2 key things. i am giving private consultation as well if you need

محمد الذوادي

I experienced the same issue when enabling the "Apply T5 Attention Mask" option, but it works fine when I disable it

Jose Cruz

Hello everyone, I'm currently trying to train a Flux LoRA model using Kohya SSon my Mac, but I'm running into an issue with the learning rate scheduler. Machine Specs: MacBook with M4 Pro Max 48 GB RAM 1000 GB storage OS: Sequoia 15.2 The Problem: Error with the Scheduler: In the logs, I see an error related to the learning rate scheduler: TypeError: CosineAnnealingLR.__init__() missing 1 required positional argument: 'T_max' I understand that for CosineAnnealingLR, the parameter T_max (which typically should be equal to the total number of training steps—for example, with 86 images and 35 epochs, roughly 3010 steps) is required so that the scheduler knows over how many steps to adjust the learning rate. This is my config: https://drive.google.com/file/d/1HUSbsO48nULLRLnRpwZT6ja6Z6px2V29/view?usp=sharing

SECourses

hello. it doesnt work on mac. i recommend you to use massed compute we have installers and coupon and everything

Nick Prestine

Hi! I am trying to follow this tutorial to fine tune flux on my linux system, I am running a dockerized version of kohya from the main repo. I specifically installed the flux 1 branch and then reinstalled kohya but I am not showing the flux 1 option when looking at the fine tune page on kohya, any insight on this?

YaoiGen

Please update this with the new de-distilled lora training method for flux-dev lora training. I am currently trying it and it's way better than training via the regular flux-dev model. Can't emphasize how much of a game-changer this new method is. You will be blown away at how beautifully it converges.

SECourses

when you give the flux dev model path it becomes enabled

YaoiGen

I feel like it deserves more attention. You put a lot of effort into testing regular flux lora training. I feel like the de-distilled needs it's own settings. Using the configs you made for regular flux lora training with de-distilled lora training didn't produce nice results. After playing around with it for a while I found that de-distilled needs different parameters. For instance, changing the Discrete Flow Shift to 3 works better than 3.1582. The learning rate needs adjustment, there is a noticeable change when using multires noise offset, and switching off t5 attention mask improves it further. All I am saying is that it needs more attention and may be worth spending time on. I was able to see convergence and good image results with only 2 epochs and even better at 6 epochs. Also prompt adherence is improved with lora's I made using de-distilled. Kohya doesn't make good samples so you need to save a lot of checkpoints and test them in swarmui, but it learns a lot faster and in fewer epochs.

SECourses

nice findings but i didnt see much improvements. also 3.1582 is the original one that was used

YaoiGen

I know, I changed it from 3.1582 to 3.0. 3.0 works better. Something must be off with the settings you are using if you aren't getting good results when inferencing in swarmui. Also, when using swarm ui you want to use the original flux-dev model, not the de-distilled model. I really think it's worth spending time on. Like I said I was able to get a character working in only 2 epochs, it's better if it's more epochs, but it was shocking how good it was.

Nick Prestine

turns out when I was attempting to reinstall kohya i was not rebuilding the container so wasn't actually updating the code based. Fixed it thank you

Nick Prestine

I am now running into another error when trying to use your 24GB_GPU_Quality_Tier_1_22950MB_14.1_Second_IT_Trains_T5_and_T5_Attention.json config file, any insight on this issue? Here is the error code: "ubuntu@grablabs-hgx-1:~/internal_comfy-cloud/Docker/dataset/images/flux$ docker logs --tail 100 kohya-ss-gui File "/home/1000/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1709, in preprocess_data processed_input.append(block.preprocess(inputs_cached)) File "/home/1000/.local/lib/python3.10/site-packages/gradio/components/dropdown.py", line 193, in preprocess choice_values = [value for _, value in self.choices] File "/home/1000/.local/lib/python3.10/site-packages/gradio/components/dropdown.py", line 193, in choice_values = [value for _, value in self.choices] ValueError: not enough values to unpack (expected 2, got 0) Traceback (most recent call last): File "/home/1000/.local/lib/python3.10/site-packages/gradio/queueing.py", line 624, in process_events response = await route_utils.call_process_api( File "/home/1000/.local/lib/python3.10/site-packages/gradio/route_utils.py", line 323, in call_process_api output = await app.get_blocks().process_api( File "/home/1000/.local/lib/python3.10/site-packages/gradio/blocks.py", line 2014, in process_api inputs = await self.preprocess_data( File "/home/1000/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1709, in preprocess_data processed_input.append(block.preprocess(inputs_cached)) File "/home/1000/.local/lib/python3.10/site-packages/gradio/components/dropdown.py", line 193, in preprocess choice_values = [value for _, value in self.choices] File "/home/1000/.local/lib/python3.10/site-packages/gradio/components/dropdown.py", line 193, in choice_values = [value for _, value in self.choices] ValueError: not enough values to unpack (expected 2, got 0) Traceback (most recent call last): File "/home/1000/.local/lib/python3.10/site-packages/gradio/queueing.py", line 624, in process_events response = await route_utils.call_process_api( File "/home/1000/.local/lib/python3.10/site-packages/gradio/route_utils.py", line 323, in call_process_api output = await app.get_blocks().process_api( File "/home/1000/.local/lib/python3.10/site-packages/gradio/blocks.py", line 2014, in process_api inputs = await self.preprocess_data( File "/home/1000/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1709, in preprocess_data processed_input.append(block.preprocess(inputs_cached)) File "/home/1000/.local/lib/python3.10/site-packages/gradio/components/dropdown.py", line 193, in preprocess choice_values = [value for _, value in self.choices] File "/home/1000/.local/lib/python3.10/site-packages/gradio/components/dropdown.py", line 193, in choice_values = [value for _, value in self.choices] ValueError: not enough values to unpack (expected 2, got 0) Traceback (most recent call last): File "/home/1000/.local/lib/python3.10/site-packages/gradio/queueing.py", line 624, in process_events response = await route_utils.call_process_api( File "/home/1000/.local/lib/python3.10/site-packages/gradio/route_utils.py", line 323, in call_process_api output = await app.get_blocks().process_api( File "/home/1000/.local/lib/python3.10/site-packages/gradio/blocks.py", line 2014, in process_api inputs = await self.preprocess_data( File "/home/1000/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1709, in preprocess_data processed_input.append(block.preprocess(inputs_cached)) File "/home/1000/.local/lib/python3.10/site-packages/gradio/components/dropdown.py", line 193, in preprocess choice_values = [value for _, value in self.choices] File "/home/1000/.local/lib/python3.10/site-packages/gradio/components/dropdown.py", line 193, in choice_values = [value for _, value in self.choices] ValueError: not enough values to unpack (expected 2, got 0) 21:09:14-071839 INFO Loading config... 21:10:03-063114 INFO Loading config... 21:18:52-068193 INFO Copy /app/data/nick/training_datasets/online_animated_objects to /app/data/nick/models/flux_online_obj_v1/img/1_vdgameobj cartoon object... 21:18:52-267194 INFO Regularization images directory is missing... not copying regularisation images... 21:18:52-268981 INFO Done creating kohya_ss training folder structure at /app/data/nick/models/flux_online_obj_v1... 21:24:14-864376 INFO Start training LoRA Flux1 ... 21:24:14-867839 INFO Validating lr scheduler arguments... 21:24:14-870828 INFO Validating optimizer arguments... 21:24:14-873184 INFO Validating lora type is Flux1 if flux1 checkbox is checked... 21:24:14-875753 INFO Validating /app/data/nick/models/flux_online_obj_v1/log existence and writability... SUCCESS 21:24:14-878867 INFO Validating /app/data/nick/models/flux_online_obj_v1/model existence and writability... SUCCESS 21:24:14-881860 INFO Validating /app/data/flux/flux1-dev.safetensors existence... SUCCESS 21:24:14-884370 INFO Validating /app/data/nick/models/flux_online_obj_v1/img existence... SUCCESS 21:24:14-887055 INFO Headless mode, skipping verification if model already exist... if model already exist it will be overwritten... 21:24:14-889978 INFO Folder 1_vdgameobj cartoon object: 1 repeats found 21:24:14-892956 INFO Folder 1_vdgameobj cartoon object: 51 images found 21:24:14-895383 INFO Folder 1_vdgameobj cartoon object: 51 * 1 = 51 steps 21:24:14-897748 INFO Regularization factor: 1 21:24:14-901542 INFO Train batch size: 1 21:24:14-903822 INFO Gradient accumulation steps: 1 21:24:14-906001 INFO Epoch: 200 21:24:14-908176 INFO max_train_steps (51 / 1 / 1 * 200 * 1) = 10200 21:24:14-910894 INFO stop_text_encoder_training = 0 21:24:14-913138 INFO lr_warmup_steps = 0 21:24:14-917625 INFO Saving training config to /app/data/nick/models/flux_online_obj_v1/model/flux_online_obj_v1_20250116- 212414.json... 21:24:14-921132 INFO Executing command: /home/1000/.local/bin/accelerate launch --dynamo_backend no --dynamo_mode default --gpu_ids 0 --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 /app/sd-scripts/flux_train_network.py --config_file /app/data/nick/models/flux_online_obj_v1/model/config_lora-20250116-212414. toml Traceback (most recent call last): File "/home/1000/.local/bin/accelerate", line 5, in from accelerate.commands.accelerate_cli import main File "/home/1000/.local/lib/python3.10/site-packages/accelerate/__init__.py", line 16, in from .accelerator import Accelerator File "/home/1000/.local/lib/python3.10/site-packages/accelerate/accelerator.py", line 32, in import torch File "/home/1000/.local/lib/python3.10/site-packages/torch/__init__.py", line 368, in from torch._C import * # noqa: F403 ImportError: /home/1000/.local/lib/python3.10/site-packages/torch/lib/../../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkComplete_12_4, version libnvJitLink.so.12 21:24:15-320882 INFO Training has ended."

Nick Prestine

disregard, fixed this by downgrading both torch and torch vision to pip install torch==2.4.1 torchvision==0.19.1

SECourses

great. i was gonna say it is environment error

Nick Prestine

Can you please send me flow that reflects what you are doing in this video FLUX Full Fine-Tuning / DreamBooth Training Master Tutorial for Windows, RunPod & Massed Compute https://www.youtube.com/watch?v=FvpWy1x5etM at time ~46 min. I am forced to use a older version of comfy that does not have the flux options like cfg etc on it and I am unsure how to mimic this process in a flow with the clip_l and t5xxl. Thank you!

SECourses

this is SwarmUI. i recommend install it and then you can see how it turns things into comfyui workflow. actually i recommend using SwarmUI over comfyui

YaoiGen

I just want to let everyone know that I discovered an interesting thing and tested it. You can literally prompt the T5xxl encoder in your captions. You just give it instructions after the captions in parentheses. So it would look like this: "A dragon flying in the sky over a town. (Pay attention to the details in the scales and notice the shape of it's wings.)" You caption regularly and then in () you give it instructions. I have tested with and without and there is a noticeable difference when training. If there is a specific detail I want the lora to learn I make sure to instruct T5 for the images that have the detail.

SECourses

wow i never tried this. when training lora you can train T5 it can be useful there

Sergio Valsecchi

Hi, I'm getting this error on MassedCompute while generating Dreambooth. Can you help me? On my local PC, I don't have any issues. Traceback (most recent call last): File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 849, in train(args) File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 197, in train ae.to(accelerator.device, dtype=weight_dtype) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1340, in to return self._apply(convert) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 900, in _apply module._apply(fn) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 900, in _apply module._apply(fn) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 900, in _apply module._apply(fn) [Previous line repeated 3 more times] File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 927, in _apply param_applied = fn(param) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1333, in convert raise NotImplementedError( NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device. Traceback (most recent call last): File "/home/Ubuntu/apps/kohya_ss/venv/bin/accelerate", line 8, in sys.exit(main()) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main args.func(args) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command simple_launcher(args) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/home/Ubuntu/apps/kohya_ss/venv/bin/python3.10', '/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py', '--config_file', '/home/Ubuntu/Downloads/Risultati/Lacoste/model/config_dreambooth-20250124-170720.toml']' returned non-zero exit status 1. 17:07:32-034530 INFO Training has ended.

SECourses

hi sorry for late reply. looks like you either given inaccurate model or didnt download model correctly: NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

Sergio Valsecchi

Strange! I ran the function "all training models" as download, and the configuration file found them as set in the download folder. I'll try again tomorrow and let you know. Thanks!

SECourses

perhaps some download were corrupted. also make sure to load fine tuning configs into dreambooth tab and lora into lora. otherwise they get corrupted permanently

SECourses

Also don't forget to run kohya updater part of the script as well

Sergio Valsecchi

Hi Furkan, thank you for the advice! I downloaded everything directly from Hugging Face, and now it's working well! It wouldn't be a bad idea if the models were included in the server image as well. Before launching the generation of LoRA or DreamBooth, it still takes 10–15 minutes. I noticed that on the server with 4 GPUs, the network goes beyond gigabit speed, while those with two GPUs reach 400 megabits, so the download obviously takes more time. https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/ae.safetensors?download=true https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/flux1-dev.safetensors?download=true https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/t5xxl_fp16.safetensors?download=true https://huggingface.co/comfyanonymous/flux_text_encoders/resolve/main/clip_l.safetensors?download=true Thanks, good work!

SECourses

you are welcome thanks for info

Sonivas Sx

2.6.0 torch available ! ! will this speed up the training even more ?

SECourses

i hope so. hopefully will test asap

Dolpin46

can u plz tell me iam fresh installing bcz iam newbie for this .and also tell me which is one Best_Configs_Better_Colors Folder . my gpu is rtx 3060 12gb so plz suggest which i go through der are new version der and old version der tq

SECourses

this config is best for you : 12GB_GPU_11100MB_9.5_second_it_Tier_1.json . make sure that you are not using more than 800 MB vram before starting training also you can compare its speed with 10GB_GPU_8700MB_14.0_second_it_Tier_1.json

Dolpin46

ok tq i will try it

Realtoonai

Hello just join the patreon today and want to train Lora. Is this Kohya_GUI_Flux_Installer_v46.zip can be used for Lora Flux, SDXL and SD 1,5 ? Should I start from here? thanks for the help

Colja Carls

I tried it with "lr_scheduler_args": "T_max=3010" (with 3010 taken as an example that you shared above) in the config.json and had no problem so far. I think in the Kohya SS GUI it is the field "LR Scheduler extra arguments", which is empty in your screenshot. Try that and tell us if it works! I'm also interested in good results (and faster results) with the lr scheduler!

Hockey

I have trained an object like a car, however it's difficult to get it to be positioned properly. For example if I want the front of the car to be in the foreground, sometimes I get the back of the car. Is this a situation where it would be beneficial to train the object with description of how it is viewed in the photo? For example: "Z3 car front", "Z3 car side" "Z3 car above" "Z3 car rear" "Z3 car front right"

SECourses

if you want such precise control makes total sense

Hockey

Thank you very much. :)

Valentyn Shumakher

I have an issue with CUDA memory when fine tuning full model, can you please check the DM's

SECourses

Replied back sorry for late reply. Discord is best to communicate

m_sil

hello, i have trained a model following your tutorial and would like to test it out in Comfy Ui but am unable to get it to work with flux. Is there a workflow you can recomend I use to do this in Comfyui?

SECourses

sadly i dont have good workflows for comfyui. that is why i recommend swarmui

Taiga

After fresh install: Traceback (most recent call last): File "E:\TRAINING\kohya_ss\kohya_gui.py", line 6, in import gradio as gr ModuleNotFoundError: No module named 'gradio'

Taiga

I have to downgrade from 3.13?

SECourses

there is no downgrade. install python 3.10 and set it as default . watch video it explains. your python 3.13 will remain functional

Z

After setting python 3.10 as default, the .bat file won't give any error but it doesn't start. It just opens for a second and closes, but nothing happens

锦涛 何

How to download the configuration file, I don't see the download link.

SECourses

you must have error somewhere else. can you do this. get lastest file. make a fresh install. when running, open a cmd and type bat file name fully and run that way. it will show error and not close.

Casper

Hey Furkan, which is the latest? Kohya_FLUX_LoRA_DreamBooth_v22.zip or Kohya_GUI_Flux_Installer_v46.zip ? What is the difference?

SECourses

just the installer. i recommend use Kohya_FLUX_LoRA_DreamBooth_v22.zip - it installs latest branch of github repo. the author combined and merged into main

Ken

Hi there Doctor, i have good results with the Kohya workflows, that you gave. I am using a 4090 for the training and the speeds are also good. I have no issues with upto Tier 3,but for Tier 2 and Tier 1, using FLux workflows, i seem to getting Flux banding, kind of like scan lines on the image whenever the image is larger than 1x in fred workflows, so above 1024 by 1024. and it alm,ost dissapears belows 1x. The images are very detailed at higher resolutions, but because of the banding/scan lines when using the Lora they are unusable.

Art of 3D Rendering

I install it here i follow your instructions and also install the Windows_RTX5000_Series_Upgrade_Run_After_Install_Finished.bat because i have a RTX 5090, but when i start a train i got always a error, any tip what may cause it? 10:46:23-348967 INFO Saving training config to D:/Ai/Train_Assets/waterline/model\flux_waterline_v1_20250502-104623.json... 10:46:23-351968 INFO Executing command: C:\AI\Flux_Train\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no --dynamo_mode default --gpu_ids 0 --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 C:/AI/Flux_Train/kohya_ss/sd-scripts/flux_train_network.py --config_file D:/Ai/Train_Assets/waterline/model/config_lora-20250502-104623.toml Could not find the bitsandbytes CUDA binary at WindowsPath('C:/AI/Flux_Train/kohya_ss/venv/lib/site-packages/bitsandbytes/libbitsandbytes_cuda128.dll') The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. Traceback (most recent call last): File "C:\AI\Flux_Train\kohya_ss\sd-scripts\flux_train_network.py", line 14, in import train_network File "C:\AI\Flux_Train\kohya_ss\sd-scripts\train_network.py", line 26, in from diffusers.models.autoencoders.autoencoder_kl import AutoencoderKL File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\diffusers\models\autoencoders\__init__.py", line 1, in from .autoencoder_asym_kl import AsymmetricAutoencoderKL File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\diffusers\models\autoencoders\autoencoder_asym_kl.py", line 22, in from ..modeling_utils import ModelMixin File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 35, in from ..quantizers import DiffusersAutoQuantizer, DiffusersQuantizer File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\diffusers\quantizers\__init__.py", line 15, in from .auto import DiffusersAutoQuantizer File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\diffusers\quantizers\auto.py", line 22, in from .bitsandbytes import BnB4BitDiffusersQuantizer, BnB8BitDiffusersQuantizer File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\diffusers\quantizers\bitsandbytes\__init__.py", line 2, in from .utils import dequantize_and_replace, dequantize_bnb_weight, replace_with_bnb_linear File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\diffusers\quantizers\bitsandbytes\utils.py", line 32, in import bitsandbytes as bnb File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\bitsandbytes\__init__.py", line 15, in from .nn import modules File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\bitsandbytes\nn\__init__.py", line 21, in from .triton_based_modules import ( File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\bitsandbytes\nn\triton_based_modules.py", line 7, in from bitsandbytes.triton.int8_matmul_mixed_dequantize import ( File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\bitsandbytes\triton\int8_matmul_mixed_dequantize.py", line 12, in from triton.ops.matmul_perf_model import early_config_prune, estimate_matmul_time ModuleNotFoundError: No module named 'triton.ops' Traceback (most recent call last): File "C:\Users\smere\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\smere\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\AI\Flux_Train\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in sys.exit(main()) File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "C:\AI\Flux_Train\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\\AI\\Flux_Train\\kohya_ss\\venv\\Scripts\\python.exe', 'C:/AI/Flux_Train/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'D:/Ai/Train_Assets/waterline/model/config_lora-20250502-104623.toml']' returned non-zero exit status 1. 10:46:30-568552 INFO Training has ended.

SECourses

testing right now with a fresh install

SECourses

yes just made an update and fixed the issue. made a fresh install and then run RTX 5000 bat file. extract new zip into folder and use it. added upgrade of bitsandbytes. use v25 zip

Art of 3D Rendering

yes it solved! in my case i just need to execute the RTX50 bat file then all working, thks

stasiu andraczek

I use RunPod with permanent network volume. I rented a cheap RTX A5000 to install everything and download all models. Everything went fine and I was able to open Kohya. I closed the pod to open a new one with four A40. Then I tried to execute Windows_Start_Kohya_SS.bat and nothing happened, I got this response: bash: @echo: command not found bash: REM: command not found bash: REM: command not found bash: gui.bat: command not found bash: REM: command not found bash: pause: command not found Do I have to make a whole installation everytime I open a new pod even with network volume? Maybe I did something wrong with my installation in the first place? I runned chmod +x RunPod_Kohya_FLUX_Installer_part1.sh ./RunPod_Kohya_FLUX_Installer_part1.sh then chmod +x RunPod_Kohya_FLUX_Installer_part2.sh ./RunPod_Kohya_FLUX_Installer_part2.sh then I copied all text from Windows_Kohya_Update.bat and executed it with terminal, and then I did the same with Windows_RTX5000_Series...Finished.bat

Alexandre Leitão

Hello, came to this patreon coz saw some ads in a reddit group about training flux lora with a 3060... where can I find that tutorial?

SECourses

hello welcome. i recommend dreambooth and if you have 64 gb ram it works great with 6 gb gpus : https://youtu.be/FvpWy1x5etM if you want lora for sure here tutorial : https://youtu.be/nySGu12Y05k for lora use config : 10GB_GPU_Quality_Tier_4_FP8_9250MB_5.8_Second_IT.json for dreambooth use : 12GB_GPU_11100MB_9.5_second_it_Tier_1.json or 10GB_GPU_8700MB_14.0_second_it_Tier_1.json make sure that you use minimum amount of VRAM before starting the training

SECourses

hi bat files are only for windows. follow this file : RunPod_Install_Instructions.txt if you are using permanent storage yes run installers everytime because it installs some of them to the local storage which is deleted everytime you change pod. it should be still faster

Alexandre Leitão

Hey man thanks for the fast repply... I'm using kohya v25.0.3... getting "too many values to unpack" when I try to load those settings for lora(10GB_GPU_Quality_Tier_4_FP8_9250MB_5.8_Second_IT.json) ... about dreambooth ... I'm still with only 32gb of ram and 12vram on those machines... running Linux... still waiting to upgrade my PC next month ... so I guess no finetuning

puk

Hello Furkan, what are the best settings for Dreambooth and Lora training with an RTX 5090?

SECourses

best dreambooth : 48GB_GPU_28200MB_6.3_second_it_Tier_1.json best lora : 48GB_GPU_Quality_Tier_3_28400MB_4.8_Second_IT.json

SECourses

32 gb is the biggest problem you have. you should upgrade to min 64 gb. too many values to unpack usually means your models were not fully downloaded or you loaded lora config into dreambooth tab at least once

Dan

Hi @secourses - I have the same issue. I tried to follow your instructions above. I reran kohya installer v27 (I did not uninstall anything first), I reran RTX5000 bat file: Q1: Should I run windows kohya update BAT first before this though? Q2: What is "extract new zip into folder and use it" I tried rerunning and it didnt work, error below: C:/Kohya_FLUX_DreamBooth_LoRA_v24/kohya_ss/sd-scripts/flux_train.py --config_file model/config_dreambooth-20250511-174547.toml Traceback (most recent call last): File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 920, in _get_module return importlib.import_module("." + module_name, self.__name__) File "C:\Python310\lib\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\loaders\single_file.py", line 24, in from .single_file_utils import ( File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\loaders\single_file_utils.py", line 28, in from ..models.modeling_utils import load_state_dict File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\models\modeling_utils.py", line 35, in from ..quantizers import DiffusersAutoQuantizer, DiffusersQuantizer File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\quantizers\__init__.py", line 15, in from .auto import DiffusersAutoQuantizer File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\quantizers\auto.py", line 22, in from .bitsandbytes import BnB4BitDiffusersQuantizer, BnB8BitDiffusersQuantizer File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\quantizers\bitsandbytes\__init__.py", line 2, in from .utils import dequantize_and_replace, dequantize_bnb_weight, replace_with_bnb_linear File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\quantizers\bitsandbytes\utils.py", line 32, in import bitsandbytes as bnb File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\bitsandbytes\__init__.py", line 15, in from .nn import modules File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\bitsandbytes\nn\__init__.py", line 21, in from .triton_based_modules import ( File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\bitsandbytes\nn\triton_based_modules.py", line 7, in from bitsandbytes.triton.int8_matmul_mixed_dequantize import ( File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\bitsandbytes\triton\int8_matmul_mixed_dequantize.py", line 12, in from triton.ops.matmul_perf_model import early_config_prune, estimate_matmul_time ModuleNotFoundError: No module named 'triton.ops' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 920, in _get_module return importlib.import_module("." + module_name, self.__name__) File "C:\Python310\lib\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion.py", line 24, in from ...loaders import FromSingleFileMixin, IPAdapterMixin, StableDiffusionLoraLoaderMixin, TextualInversionLoaderMixin File "", line 1075, in _handle_fromlist File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 910, in __getattr__ module = self._get_module(self._class_to_module[name]) File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 922, in _get_module raise RuntimeError( RuntimeError: Failed to import diffusers.loaders.single_file because of the following error (look up to see its traceback): No module named 'triton.ops' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\sd-scripts\flux_train.py", line 33, in from library import deepspeed_utils, flux_train_utils, flux_utils, strategy_base, strategy_flux File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\sd-scripts\library\flux_train_utils.py", line 17, in from library import flux_models, flux_utils, strategy_base, train_util File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\sd-scripts\library\train_util.py", line 48, in from diffusers import ( File "", line 1075, in _handle_fromlist File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 911, in __getattr__ value = getattr(module, name) File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 911, in __getattr__ value = getattr(module, name) File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 910, in __getattr__ module = self._get_module(self._class_to_module[name]) File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\diffusers\utils\import_utils.py", line 922, in _get_module raise RuntimeError( RuntimeError: Failed to import diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion because of the following error (look up to see its traceback): Failed to import diffusers.loaders.single_file because of the following error (look up to see its traceback): No module named 'triton.ops' Traceback (most recent call last): File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in sys.exit(main()) File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "C:\Kohya_FLUX_DreamBooth_LoRA_v24\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\\Kohya_FLUX_DreamBooth_LoRA_v24\\kohya_ss\\venv\\Scripts\\python.exe', 'C:/Kohya_FLUX_DreamBooth_LoRA_v24/kohya_ss/sd-scripts/flux_train.py', '--config_file', 'model/config_dreambooth-20250511-174547.toml']' returned non-zero exit status 1. 17:46:16-738250 INFO Training has ended.

SECourses

hi do this. get latest zip file. install latest zip file fresh install. then use rtx 5000 series upgrade. it should fix

Dan

Hi there, please, 1. a fresh install = delete all old files in kohya? (My system has CUDA 12.4 as default...) ? 2. Do I also run 'Windows_Kohya_Update'?

SECourses

yes delete all. when you do fresh install you dont need windows update kohya. just do fresh install and then use rtx 5000 series upgrade bat file also have cuda 12.8 in your system and make it default

SECourses

also have cuda 12.8 in your system and make it default

Petz

Hi, thanks for your Tutorials! I followed the one to install Kohya and i wanted to start a Training run with one of your configurations. I am getting the following message, maybe you can help:Traceback (most recent call last): File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\sd-scripts\train_network.py", line 1884, in trainer.train(args) File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\sd-scripts\train_network.py", line 589, in train model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator) File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\sd-scripts\train_network.py", line 179, in load_target_model text_encoder, vae, unet, _ = train_util.load_target_model(args, weight_dtype, accelerator) File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\sd-scripts\library\train_util.py", line 5480, in load_target_model text_encoder, vae, unet, load_stable_diffusion_format = _load_target_model( File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\sd-scripts\library\train_util.py", line 5435, in _load_target_model text_encoder, vae, unet = model_util.load_models_from_stable_diffusion_checkpoint( File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\sd-scripts\library\model_util.py", line 1000, in load_models_from_stable_diffusion_checkpoint _, state_dict = load_checkpoint_with_text_encoder_conversion(ckpt_path, device) File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\sd-scripts\library\model_util.py", line 977, in load_checkpoint_with_text_encoder_conversion checkpoint = torch.load(ckpt_path, map_location=device) File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\venv\lib\site-packages\torch\serialization.py", line 1548, in load raise pickle.UnpicklingError(_get_wo_message(str(e))) from None _pickle.UnpicklingError: Weights only load failed. In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. Please file an issue with the following so that we can make `weights_only=True` compatible with your use case: WeightsUnpickler error: Unsupported operand 192 Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. Traceback (most recent call last): File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in sys.exit(main()) File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main args.func(args) File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command simple_launcher(args) File "F:\Programme\Kohya_FLUX_DreamBooth_LoRA_v28\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['F:\\Programme\\Kohya_FLUX_DreamBooth_LoRA_v28\\kohya_ss\\venv\\Scripts\\python.exe', 'F:/Programme/Kohya_FLUX_DreamBooth_LoRA_v28/kohya_ss/sd-scripts/train_network.py', '--config_file', 'F:\\Programme\\Lora_Training\\Neuer Ordner (3)\\model/config_lora-20250519-102357.toml']' returned non-zero exit status 1. 10:24:22-233429 INFO Training has ended.

SECourses

hi your model download failed for some reason. so your model corrupted. delete model and redownload

Bas s

Do you have Kohya settings for training an object? Like a purse, cellphone, etc.... or will using the latest flux lora presets work for that?

SECourses

all presets works. only dataset changes slightly you need a better dataset.

Bas s

If my dataset is not too big will incrementing the repeats work?

SECourses

actually you should reduce epoch count if dataset is too big

洋次郎 山崎

I've seen articles that recommend using dev2pro for Flux LoRA and fine-tuning training. Have you experimented with it? I would appreciate your perspective on this. Additionally, do you recommend using the official dev model?

arquinovatos (edited)

Comment edits

2026-09-07 10:00 in the video: "Blazing Fast & Ultra Cheap FLUX LoRA Training on Massed Compute & RunPod Tutorial - No GPU Required!" You said at minute 22:30 that I could disable t5 attention to hugely speed up training loosing some quality, but I can't find that field in the current version of Kohya. Was this removed in an update?
2025-07-12 21:56

SECourses

well i can see maybe you cant find open all tabs

SECourses

are you on massed compute or runpod?

SECourses

i told you you have to give flux base model path accurately first then it will refresh and appear

SECourses

ok in that case get a new machine and use latest installer file. i updated gradio version it can fix. also you are on runpod or massed compute?

SECourses

ok i also started a new machine to test

SECourses

L40 is weak. I recommend RTX A6000 ADA

SECourses

sorry for late reply did you proceed and run? it should run

arquinovatos (edited)

Comment edits

2026-09-07 10:00 I was in the middle of a training in massed compute and I had to restart my local PC. When I connected again I noticed it was closed as if the massed compute PC restarted. Is this normal?
2025-07-13 03:05

SECourses

it was because you selected end current session - i told this numerous times in tutorials even in last one but probably you missed. it kills all the running apps

Hazard

Whats the best steps and epoch for style lora training?

Hazard

is multi gpu training not faster when training sdxl ?

v. svobodov

Hello. Thank you for your work, it has helped me a lot. I want to run a Lora training on Massed Compute. I’m having trouble following your instructions. Before starting the training, I tried to upload a file to the Huge Face repository, but I’m getting this error. I’ve tried multiple options, and this seems to be the last one. Could you please tell me what might be causing this issue? ///////////////////////////////////////////////////////////////////////////////////////// """Upload a large folder to the Hub in the most resilient way possible. ■ Stable-Diffusion 14m ago 5255 5256 Several workers are started to upload files in an optimized way. Before being committed to a repo, files must be style_models ignore_patterns-ignore_patterns, num_workers=num_workers, print_report=print_report, print_report_every=print_report_every, File -/apps/venv/lib/python3.10/site-packages/hugging face_hub/_upload_large_folder.py:84, in upload_large_folder_internal (api, repo_id, folder_path, repo_type, revision, private, allow patterns, ignore_patterns, num_workers, print_report, print_report_every) 82 folder_path = Path (folder_path).expanduser().resolve() 83 if not folder_path.is_dir(): ---> 84 raise ValueError(f "Provided path: '{folder_path}' is not a directory") 86 if ignore_patterns is None: 87 ignore_patterns = [] ValueError: Provided path: '/apps/Stable SwarmUI/Models/Lora' is not a directory Upload a single file with specific name to remote repo - Wait till UPLOAD COMPLETED printed [ ]: # This cell is used to upload single file into a repo with certain name from hugging face_hub import HfApi api = HfApi() api.upload_file( path_or_fileobj=r"/home/ubuntu/apps/stable-diffusion-webui/models/Stable-diffusion/model_name.safetensors", path_in_repo="model_name.safetensors"

SECourses

it is. when you do multi gpu you need to reduce epoch count with number of gpus. so 100 epoch 2x gpu = 1 x gpu 200 epochs

v. svobodov

First of all, thank you for your prompt reply. As an instruction, I used this video "FLUX LoRA Training Simplified: From Zero to Hero with Kohya SS GUI" And yes, including the one you specified. As an instruction, I relied on this video. The problem arose with the Huge Face. Before training Lora, I wanted to check if I could send the Lora files to the repository on the Huge Face. To do this, I logged into a remote Massad Compute computer. I downloaded Hugging_Face_Upload_Download_V15.ipynb to it and started following the instructions step by step. The first two steps were successful, but the rest had problems with the directory from where I tried to upload Lora files. (this directory is /apps/StableSwarmUI/Models/Lora), I've tried different variations... I uploaded several of my Lora files uploaded to Massive Compute to this folder, as an example, and tried to upload them to Huge Face. This is where the error occurred, judging by the inscription with the "directory". Example of a step in which an error: ("cell_type": "code", "execution_count": null, "id": "30b96a4f", "metadata": {}, "outputs": [], "source": [ "!huggingface-cli upload-large-folder \"YourUserName/reponame\" \"/home/Ubuntu/apps/StableSwarmUI/Models/diffusion_models\" --repo-type=model --no-bars\n", "\n", "print(\".\\n.\\nUPLOAD COMPLETED\") I tried to upload the Lora files from the already pre-installed on Massed Compute - SwarmUI, maybe the problem with the directory is this. I repeat, I wanted to check before starting the Lora training whether I would be able to output the Lora files from a remote computer at the final stage, because only this moment remains unclear to me, I have more or less mastered and reproduced all the other stages. A possible solution, it seems to me, would be to follow all the instructions in your this video "FLUX LoRA Training Simplified: From Zero to Hero with Kohya SS GUI" step by step. - after uploading everything necessary for training to the "download" folder on the remote computer, specify this directory after the end of the training, as it seems to me, then this error that I encountered will be fixed. What do you think?

SECourses

what you are trying to upload or download? because i see you didnt give your own hugging face repo name here : YourUserName/reponame

v. svobodov

The example I provided is just an example to make it clear at which step the error occurred, copied from your file: Hugging_Face_Upload_Download_V15.ipynb. "hugging face repo name here : YourUserName/reponame" - I entered my own, the correct one taken from the Huge Face. I tried to upload Loras files that I had pre-uploaded to /apps/StableSwarmUI/Models/Lora to the Massed Compute VM. I wanted to check if I could upload Lora files after training to avoid a situation in which I had trained my Loras and could not send Lora files from the Massed Compute VM.

v. svobodov

And at the above step, I get an error related to the directory from where I tried to upload Lora files. I can send a screenshot of the screen that will clearly show what happened, but i can't upload the screenshot here.

v. svobodov

The problem has been solved.

geddon

Is it possible to train Kontext LoRAs with Kohya_ss?

SECourses

he added that into musubi tuner recently

SECourses

great thanks for info sorry for late reply

SECourses

well it says you the error. ValueError: Provided path: '/apps/Stable SwarmUI/Models/Lora' is not a directory massed compute always starts with /home/ubuntu/ so accurate path is /home/ubuntu/apps/Stable SwarmUI/Models/Lora

DavidO

Hi thank you for this tutorial, I very successfully trained flux several times with this technique and your presets, I am interested in trying it with flux krea, on a RTX590, I wondered if you had any plans to explore this?

SECourses

it works right away with our workflow. just change base model to krea. it is tested and verified.

DavidO

Thank you for the fast reply, I had tried it but krea does not show up when the preset is applied, I will have another look thanks again

SECourses

load normal preset and just change model path.

Yvan Nost

Thanks for this tutorial. I learned a loooot ! 🥳 I am training a style, fully captioned by hand, my dataset contains 247 images. Should I still aim for 200 epochs ? It seems like a lot

SECourses

probably you dont need. when i used 256 images my best epoch was 70. even 50 can be sufficient

Yvan Nost

Thanks for your quick answer ! Can we maybe chat somewhere quickly ? I won't bother you too long 😊

SECourses

you can message from discord. i am also giving private lectures

Yvan Nost

Thanks, I just joined your discord server and sent you a friend request. #nvd3227 Maybe a private lecture will be needed if I can't fix my problem quickly :)

Ariloum

Hello, I'm having some strange issue with result character loras - all of them are producing just latent image while generating with FLux (at low lora weights like 0.2-0.8 it just didn't catch the character and after 0.8-1 image start noising and just becomes latent image if I set weight to 1.5-2). I'm using latest kohya, latest sd-scripts, and 24GB VRAM configs (tried them all) on my RTX 5090. I tried also many different options to change (tried fp8 and fp16 both flux model and text encoder in all combinations), including full reinstall of kohya and trying to make training inside ComfyUI with the ComfyUI-FluxTrainer node.

Ariloum

I don't think it's dataset related, I have lots of datasets that already trained with sd15/sdxl/flux (maybe it's over 30) and in case of flux I have ~6 datasets that I trained with some older build of kohya (that I used in March month). I' experiencing some strange behave there with my RTX 5090 and flux train in kohya - it often cause my monitor randomly turns off or it shows latent noise all over the screen and PC stuck so I have to reboot it. I don't have any other issues with high load while generating flux+loras content or wan wideos or gaming e.t.c.. Curiously that if I'm using ComfyUI-FluxTrainer node to train lora that bug never happens but trained lora's I'm training are broken too. And I'm kinda lost on this issue - have no idea what could cause this. I tried to monitor GPU temperatures/throttling - but it's never go over 73C for memory and much lower for the core on full load. I tried reinstalling both drivers studio/gaming. Also I tried to undervolt the GPU and decrease memory/core frequencies but it changes nothing.

SECourses

ComfyUI-FluxTrainer could be reason

Ariloum

both kohya_ss and ComfyUI-FluxTrainer are using same "sd-scripts" trainer. Actually you don't need not kohya_ss neither ComfyUI-FluxTrainer for training (both just UI), you can run just a script in a batch file referring to the acceleration and passing params/configs to the sd-scripts. And there is some conflict on my side with sd-scripts/configs and my GPU/system which I can't resolve yet

Ariloum

Today I tried to run kohya in the docker container with ubuntu 22 on my Windows 10 with using your training images dataset, training process finished fine but result LoRA's has same glitch and didn't catch the person at all. I tried few train attempts with 300-1000 steps and different flux model/text encoder combinations - like mixing fluxdev-fp8/fluxdev-bf16 with t5xxl_fp8/t5xxl_fp16 . Result images generated in comfyui are not even close to the dataset - here is some image example with 1000 steps LoRA (fluxdev-bf16+t5xxl_fp16) and weight of 1: https://www.dropbox.com/scl/fi/v6nvfvkvp5girhkfy5q3f/20250921-14-59-03-ComfyUI_00001_.png?rlkey=bx1pisy29mpnk5cpooi8dwxm5&st=4ph2kg91&dl=0 By the way, I used this config mostly: 24GB_GPU_Quality_Tier_4_FP8_17250MB_4.9_Second_IT.json and I see very big numbers on avr_loss=0.45 - shouldn't this loss equals like 25%?

SECourses

big loss number totally ok i get same. sadly i dont know what you are doing wrong but i just did a training recently it just works perfect.

SECourses

sadly i dont know what you are doing wrong but working perfect for me. if you need private consultation i give it too. 125$ for individuals and 250$ for companies. you would see how well it will work :D

Ariloum

1. I will try one more time fresh setup from scratch 2. If I accept private consultation but it still won't work will I have to pay? 3. by the way Ubuntu22 + Docker - setup I did - is as fast as 3s/it while your setup files on bare windows is as slow as 5-7s+/it (I used same kohya training config for both)

SECourses

windows could be a little bit slow but it shouldnt be that much difference. i can only guarantee work will happen on runpod or massed compute i dont know your docker setup.

Ariloum

That docker build used latest CUDA 12.8.1 and maybe other libs too (I haven't checked, just made that config with help of the Claude) I don't need runpod/massed compute, I want it to run locally, but for some reason all builds I'm trying to run fails to catch character from the training data. It have been worked in May, and a few weeks ago I succesfully trained 1 LoRA with your config and then something broke and I can't make it work anymore. I can share you docker setup if you interested in

Ariloum

I just successfully trained flux LoRA with my docker kohya build (and it was lightning fast ending with 2.87s/it), I think it will work well with usual windows builds as well. the problem was with the RTX 5090 and fp8/bf16 flux+encoder and your config 24GB_GPU_Quality_Tier_4_FP8_17250MB_4.9_Second_IT.json - it's not working at all in any combination of models on these video cards. it all goes well when I switched to bigger bf16 flux model and another config 24GB_GPU_Quality_Tier_3_23150MB_5.9_Second_IT.json, now I didn't get the monitor glitch and LoRA's represents the character also it's hilarious that under docker's ubuntu I'm getting like twice faster training speed on windows with ~3s/it, screenshot: https://tinyurl.com/ysr2uwbk

Ariloum

just tested - on windows it's 4.54s/it (same config, same settings/models and dataset), so for 3000 steps it's 3h30min instead of 2h20min - 50% slower I'm going to make another docker container for comfyui to test generation speed

SECourses

you know that big diff telling me that is it not accurately working in your docker. that is why faster

SECourses

wait didnt you say windows results are bad compared to linux? you get same results but different training speed?

Ariloum

no, I said your config 24GB_GPU_Quality_Tier_4_FP8_17250MB_4.9_Second_IT.json is not working for me on RTX 5090 on these environments: windows-kohya, windows-comfyui-FluxTrainer, ubuntu-docker-kohya (with that config LoRA's didn't catch dataset at all no matter how many steps 300 or 6000 and I also had monitor glitches/pc hangs with that) I make it working with tier 3 config, but overall - between windows and docker-ubuntu - training is 50% faster on docker ubuntu: 3000 steps takes 140 mins on docker-ubuntu (under windows host) and 210 mins on bare windows, so I will train all my LoRA's now under docker-linux because it's much faster.

SECourses

ping me from discord i will test 5090 on runpod and on my pc and share screenshots. more people can comment too

Sergio Valsecchi

I tried to install v31 on Massed Compute with an RTX6000 96GB. After following all steps as before, I ran: pip install huggingface_hub requests hf_xet chmod +x Massed_Compute_Kohya_FLUX.sh ./Massed_Compute_Kohya_FLUX.sh But before finishing the update installation I get this error: Warning: LD_LIBRARY_PATH environment variable is not set. Certain functionalities may not work correctly. Please ensure that the required libraries are properly configured. If you use WSL2 you may want to: export LD_LIBRARY_PATH=/usr/lib/wsl/lib/ Any advice? grazie sergio

SECourses

yep that is deliberate. just proceed it works i tested.

Sergio Valsecchi

Hi Furkan, maybe I didn’t express myself well—let me try again. I spin up an instance on Massed Compute and try to start kohya_ss, and it works. After running the update v31 with pip install huggingface_hub requests hf_xet chmod +x RunPod_Install_Kohya.sh ./RunPod_Install_Kohya.sh the error I get is Warning: LD_LIBRARY_PATH environment variable is not set. Certain functionalities may not work correctly. Please ensure that the required libraries are properly configured. If you use WSL2 you may want to: export LD_LIBRARY_PATH=/usr/lib/wsl/lib/ Traceback (most recent call last): File "/home/Ubuntu/apps/kohya_ss/kohya_gui.py", line 6, in import gradio as gr ModuleNotFoundError: No module named 'gradio' Ubuntu@0044-kci-prxmx10102:~/Downloads/V31$ After that, even running cd /home/Ubuntu/apps/kohya_ss source venv/bin/activate unset LD_LIBRARY_PATH ./gui.sh --listen=0.0.0.0 --inbrowser --noverify gives me the same error. You say it works, but I can’t find a way to get kohya_ss running again other than doing a downgrade. Do you have any other details I can follow? Sergio

SECourses

if you run chmod +x RunPod_Install_Kohya.sh ./RunPod_Install_Kohya.sh this on massed compute wont work you need to run this : pip install huggingface_hub requests hf_xet chmod +x Massed_Compute_Kohya_FLUX.sh ./Massed_Compute_Kohya_FLUX.sh

Leon Emmanuel ISHIMWE

After following the instruction for v31 on Massed Compute A6000, it's saying gradio not found after executing everything in the massed compute install instructions.

SECourses

ye that means install error somewhere. can you try again and send me install logs : monstermmorpg@gmail.com

Zeeshan Gilani

hello everyone, I am getting the same error, it'll be helpful if you can provide a solution, thanks

SECourses

the error means install error. did you follow Massed_Compute_Kohya_FLUX_Instructions.txt ? please send me cmd logs of install and run : monstermmorpg@gmail.com

Leon Emmanuel ISHIMWE

I ended up activating the env, and installed the requirements.txt inside the zip folder. It seems like the install script skips that step.

SECourses

install script installs requirements.txt weird. hjere code $SCRIPT_START_DIR is your beginning folder cd "$SCRIPT_START_DIR" pip install -r requirements.txt cd /home/Ubuntu/apps cd kohya_ss cd sd-scripts pip install -r requirements.txt cd .. cd .. cd "$SCRIPT_START_DIR" python3 setup_accelerate.py cd /home/Ubuntu/apps/kohya_ss source venv/bin/activate unset LD_LIBRARY_PATH ./gui.sh --listen=0.0.0.0 --inbrowser --noverify

Sergio Valsecchi

Yes, sorry — I misunderstood and pasted the wrong thing. Yesterday I tried again with: pip install huggingface_hub requests hf_xet chmod +x Massed_Compute_Kohya_FLUX.sh ./Massed_Compute_Kohya_FLUX.sh but I still can’t figure out why I keep getting the same error. I’ll try again today, but reading through the comments, I see I’m not the only one having trouble getting it to run — while on Windows, on my PC, it works perfectly.

SECourses

hi i made a tutorial just for you please follow and use v32 zip file : https://youtu.be/Ym9rdfy2VZ0 make sure to select SECourses image on massed compute with SECourses coupon

Floppe

Hi, I keep getting this error every time I start the training. Could you tell me what I’m doing wrong? Thanks a lot for the great tutorial. Best, Floppe

Floppe

enable full bf16 training. Traceback (most recent call last): File "E:\koya\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\sd-scripts\flux_train_network.py", line 547, in trainer.train(args) File "E:\koya\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\sd-scripts\train_network.py", line 880, in train unet = self.prepare_unet_with_accelerator(args, accelerator, unet) # accelerator does some magic here File "E:\koya\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\sd-scripts\flux_train_network.py", line 518, in prepare_unet_with_accelerator accelerator.unwrap_model(flux).prepare_block_swap_before_forward() File "E:\koya\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\sd-scripts\library\flux_models.py", line 1009, in prepare_block_swap_before_forward self.offloader_double.prepare_block_devices_before_forward(self.double_blocks) File "E:\koya\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 246, in prepare_block_devices_before_forward weighs_to_device(b, torch.device("cpu")) # make sure weights are on cpu File "E:\koya\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\sd-scripts\library\custom_offloading_utils.py", line 106, in weighs_to_device module.weight.data = module.weight.data.to(device, non_blocking=True) torch.AcceleratorError: CUDA error: resource already mapped CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. Traceback (most recent call last): File "C:\Users\Flori\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\Flori\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "E:\koya\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 6, in sys.exit(main()) File "E:\koya\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 50, in main args.func(args) File "E:\koya\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1213, in launch_command simple_launcher(args) File "E:\koya\Kohya_FLUX_DreamBooth_LoRA_v31\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 795, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['E:\\koya\\Kohya_FLUX_DreamBooth_LoRA_v31\\kohya_ss\\venv\\Scripts\\python.exe', 'E:/koya/Kohya_FLUX_DreamBooth_LoRA_v31/kohya_ss/sd-scripts/flux_train_network.py', '--config_file', 'E:/koya/Kohya_FLUX_DreamBooth_LoRA_v31/training/training_run/training_ohwx_lora_flux\\model/config_lora-20251010-224918.toml']' returned non-zero exit status 1.

SECourses

hi what is your gpu, how much RAM you have, which preset you are using? exact name of preset

Floppe

Hi, I have an RTX 4070 GPU and I’ve tested all of the following presets, but the same error occurs every time: 12GB_GPU_Quality_Tier_2_11250MB_14.3_Second_IT_and_T5_Attention 12GB_GPU_Quality_Tier_3_11150MB_15.3_Second_IT 10GB_GPU_Quality_Tier_3_FP8_9050MB_11.8_Second_IT_and_T5_Attention 10GB_GPU_Quality_Tier_3_8800MB_11.1_Second_IT

SECourses

ok but what is your RAM amount? 4070 is good gpu. if your ram is 32 GB you can try 32 GB RAM Configs - Not VRAM - RAM folder but 32 GB RAM is a big problem. upgrade 64 GB and it should fix immediately

Floppe

Alright, I actually only have 32 GB of RAM at the moment, so I'll go ahead and upgrade to 64 GB first and then test again. Thanks.

SECourses

awesome let me know after that. i am 100% sure it will work hopefully

Floppe

Hi, so with 64GB RAM the configs are working, but unfortunately I'm currently unable to prevent an overflow into the shared GPU memory. Even when only 10GB of VRAM are being used, I still see about 13GB of shared GPU memory in use. Is there any way to optimize this further? I have tested all 10GB and 12GB VRAM profiles. Best regards and thanks for your help.

SECourses

yes that is totally expected but it is actually not shared VRAM. it is doing block swapping. normally this model takes 28 gb VRAM. what is your step speed?

Floppe

22-28 s

SECourses

that sounds like ok it should get faster. currently rtx 3090 TI is like 10 second / it

Floppe

OK, thanks for your feedback.

MR

Hi, I have a MacBook Pro m4 max with 64gb, can I do this on it? I’m having trouble finding comfyui material for it, can you tell me a workflow for flux image generation and how to train a lora. If you have videos, show me which ones I should watch. Thank you very much.

SECourses

i don't think so you can train on mac sadly. so train on Massed Compute or RunPod then download models and use on your mac with ComfyUI or SwarmUI

Dmitrii

do you plan to backport torch compile and fp8 scaled training to sd-scripts?

manoj ramanchira

Hello Dr. In your config file '32GB_GPUs_26.7GB_VRAM_Quality_1.json' the number of epochs is 200. This is for how many images? Also in your article, you mentioned 70 epochs for 256 images. So should I stick to 70 epochs & batch size =1 for 256 images ? Also I see that 'Enable bucket' is NOT enabled in the config file. Is it because you have only one aspect ratio for all the data set? Thanks

SECourses

well 200 epochs up to 50 images working great. for 256 images i recommend up to 100 epochs but 70 can be sufficient too. yes i dont enable bucketing because you should preprocess verify and then enable to be sure.

Dmitrii

I found it only in fork here: FurkanGozukara/sd-scripts-premium, but not in original kohya-ss/sd-scripts

manoj ramanchira

Thaks Dr. Few more questions: 1) For 256 image data set & for an epoch value of 100, what should be the optimal Learning Rate for both LoRa and Dreambooth? I'm aiming for Quality 1 on a 5090 GPU 2) What' the significance of 'Min SNR gamma'. Should the value be 0 or 5? 3) Should we enable 'Text Encoder learning rate'? 4) Do we need to change the value of 'Max num workers for DataLoader' to say 8? Will it improve speed? Or just leave it to 0?

SECourses

learning rate same. dont change others.

SECourses

yes that is what i am developing for you. sd scripts dont have it. i coded it

manoj ramanchira

Thanks. For Dreambooth training, what's the ideal dataset count? For LoRa, I've 256 images. Should I use same number of images for Dreambooth training as well, or have a much less image count? Thanks

SECourses

no same strategy. actually i had my best results with 256 images dreambooth

manoj ramanchira

Cool, then. Thanks. Let me try. Hopefully, my system could handle Dreambooth traning..

Leonid Wolf

Good time of day. Please tell me, how can I eliminate the terribly slow caching during training preparation?

SECourses

the caching must run on GPU. make sure that. also you can increase vae batch size. it is batch size of caching. what gpu preset you using? if it is set to CPU it will be super slow

Caleb

Thank you for these, I really appreciate it. I have recently switched to ubuntu on local, I have tried to use the install from runpod scripts and I keep getting an out of memory error with the 24gb presets. Do you have any advice for which config I should use on a local ubuntu install?

SECourses

well it definitely should not happen. how much RAM you have? are you following your RAM? did you set virtual RAM on Linux?