Home
Creators
Search Recent Random
Posts
Search DMs Popular Hash Lookup Tags Random
Importer
Import FAQ
Register Favorites Keys Review DMs Logout
Community
Telegram ThePornDude
Home Creators Posts Import Register Favorites Logout

For RunPod - Automatic Kohya SS LoRA Installer (Patreon)

Published:
2023-06-21 10:08:03
Edited:
2025-05-25 11:54:49
Imported:
2026-06
Tags:

Downloads

Content

Patreon exclusive posts index to find our scripts easily, Patreon scripts updates history to see which updates arrived to which scripts and amazing Patreon special generative scripts list that you can use in any of your task.

Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord

Please also Star, Watch and Fork our Stable Diffusion & Generative AI  GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)

=======

RunPod Automatic1111 updater : https://www.patreon.com/posts/1-click-runpod-86438018

WARNING USE THIS INSTALLER HERE : https://www.patreon.com/posts/112099700

USE THIS INSTALLER HERE : https://www.patreon.com/posts/112099700

PREVIOUS INSTALLERS ARE OUTDATED SO USE ABOVE INSTALLER

THIS ABOVE INSTALLER SUPPORTS ALL SD 1.5, SD 3.5, SDXL, FLUX, ETC

Best Configs

For best SD 1.5 Kohya config Click here, and for best SDXL Kohya config Click here

23 May 2024 Update

For 8400 (200 repeat) steps training on RunPod, full configuration image : runpod_8400_Steps.png

For Windows only folder paths would change

18 April 2024 Update

kohya_installer.sh updated to the latest version

Updated 12 November 2023:

downloadSDXL.sh will now download SDXL FP16 VAE so you don't need --no-half-vae anymore

It will also download SDXL FP32 VAE which you can use during training

chmod +x downloadSDXL.sh

./downloadSDXL.sh

Updated 10 September 2023:

Execute below code 1 time after each restart of the Pod to get super insane speed with A5000 GPU. Same price of RTX 3090. 1.64 IT/s for 1024x1024 full SDLX DreamBooth training

apt update

apt install libcudnn8=8.9.7.29-1+cuda11.8 libcudnn8-dev=8.9.7.29-1+cuda11.8

Updated 11 August 2023

To use certain GPU if you have multiple GPUs : --gpu_ids=1

Changes : It will auto download Realistic_Vision_V5.1.safetensors and vae-ft-mse-840000-ema-pruned.ckpt (best VAE for SD 1.5 models) and sdxl_vae.safetensors (best VAE for SDXL) into the relative folders in both templates.

Best SDXL LoRA Training Tutorial published 8 August 2023 > Become A Master Of SDXL Training With Kohya SS LoRAs - Combine Power Of Automatic1111 & SDXL LoRAs

Instructions

downloadSDXL. sh file will automatically download latest models mentioned above

RunPod io link > https://bit.ly/RunPodIO

* Select stable-diffusion:web-ui or stable-diffusion:fast-stable-diffusion templates

* Make container disk size at least 15 GB

Download attached kohya_installer.sh, downloadSDXL.sh and then upload it into your Workspace folder

Screenshot of upload

Click here to open install commands on GitHub

Or here quick command

chmod +x kohya_installer.sh

./kohya_installer.sh

If you pick stable-diffusion:web-ui template download attached relauncher.py, enter inside stable-diffusion-webui folder, upload relauncher. py. It will ask you to Overwrite so overwrite. Then restart your pod 1 time. This is only necessary 1 time to prevent permanent relaunching of Automatic1111 Web UI.

You can also use this runpodctl auto installer from this Patreon post > https://www.patreon.com/posts/1-click-bat-file-87505171

Check section Auto Installer For RunPod

Screenshot of install commands

Use a custom model otherwise not working.

.e.g : /workspace/stable-diffusion-webui/models/Stable-diffusion/v1-5-pruned.ckpt

Screenshot of custom model select

Use below command to kill Web UI instance to save more VRAM before doing Kohya Training

fuser -k 3000/tcp

Use below command to start Kohya on RunPod after installation

apt update 

yes | apt-get install python3.10-tk

fuser -k 7860/tcp

cd /workspace/kohya_ss

source venv/bin/activate

export CUDA_VISIBLE_DEVICES=0

bash gui.sh --share --headless

By changing CUDA_VISIBLE_DEVICES  to different CUDA you can start different training on each GPU

SDXL training presets : https://www.patreon.com/posts/very-best-for-of-89213064

-

-

Files

Comments

John Vee

hello can we run this auto installer with Automatic1111 as well? I tried it but it broke Automatic1111. However your manual install works fine because of venv thanks

John Vee

yes sir I'm using for Runpod. I had Automatic1111 installed first, and then I tried this autoinstaller, but it didn't work for me. thanks

John Vee

yes sir I follow your instructions to create runpod I'm using the latest runpod/stable-diffusion:web-automatic-8.0.3 template

John Vee

oh I think I mis-understood - you want us to use RunPod Fast Stable Diffusion runpod/stable-diffusion :fast-stable-diffusion-2.1.0

John Vee

ok thank you

John Vee

thank you sir! it makes your member's lives easier to have the auto scripts working together! I appreciate your time and wisdom

John Vee

yes I just tested and it's perfect thanks again sir!

Jacek Jabłoński

kohya_installer.sh & kohya_restart.sh "Attachment was not found"

Ihor Liulchuk

hello! can you help me? I installed, actually reinstalled Kohya. started training and here we are, nothing happens further... 18:01:52-246696 INFO Copy /workspace/kohya_ss/image to /workspace/kohya_ss/test1/img/40_ohwx cartoon... 18:01:52-499795 INFO Copy /workspace/kohya_ss/image to /workspace/kohya_ss/test1/reg/1_cartoon... 18:01:52-706080 INFO Done creating kohya_ss training folder structure at /workspace/kohya_ss/test1... 18:01:58-758332 INFO Start training LoRA Standard ... that's all

Ihor Liulchuk

I have this issue when I clicked to downolad here kohya_installer.sh {"errors":[{"code":902,"code_name":"AttachmentNotFound","detail":"Attachment with id 14718202 was not found.","id":"0c474eac-7ab5-5713-bd26-988d2521663c","status":"404","title":"Attachment was not found."}]}

Nakul Kapdi

08:14:23-266756 INFO Version: v21.8.2 08:14:23-277637 INFO nVidia toolkit detected 08:14:31-720252 INFO Torch 2.0.1+cu118 08:14:31-749670 ERROR Could not load torch: cuDNN version incompatibility: PyTorch was compiled against (8, 7, 0) but found runtime version (8, 5, 0). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN.one possibility is that there is a conflicting cuDNN in LD_LIBRARY_PATH. root@f33c9177229a:/workspace#

Nakul Kapdi

Hello It is working now , but when i am creating image caption using kohya_ss ->Utiliti -> Blip Caption this error coming help me on this => ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ RuntimeError: The size of tensor a (48) must match the size of tensor b (576) at non-singleton dimension 0 08:59:55-467149 INFO ...captioning done

Rise Above The Machines

I keep getting this eroror root@b65b9088f02d:/workspace# apt update yes | apt-get install python3.10-tk fuser -k 7860/tcp cd /workspace/kohya_ss source venv/bin/activate pip install fastapi==0.99.1 bash gui.sh --share --headless Hit:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease Hit:2 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease Hit:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease Reading package lists... Done Building dependency tree... Done Reading state information... Done 51 packages can be upgraded. Run 'apt list --upgradable' to see them. Reading package lists... Done Building dependency tree... Done Reading state information... Done Note, selecting 'python3-tk' instead of 'python3.10-tk' python3-tk is already the newest version (3.10.8-1~22.04). 0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. Cannot stat file /proc/43/fd/0: Permission denied Cannot stat file /proc/43/fd/1: Permission denied Cannot stat file /proc/43/fd/2: Permission denied Cannot stat file /proc/43/fd/4: Permission denied Cannot stat file /proc/43/fd/5: Permission denied Cannot stat file /proc/43/fd/6: Permission denied Cannot stat file /proc/43/fd/7: Permission denied Cannot stat file /proc/43/fd/8: Permission denied Cannot stat file /proc/43/fd/9: Permission denied Cannot stat file /proc/43/fd/10: Permission denied Cannot stat file /proc/43/fd/11: Permission denied Cannot stat file /proc/43/fd/12: Permission denied Cannot stat file /proc/43/fd/13: Permission denied 7860/tcp: 1255 Requirement already satisfied: fastapi==0.99.1 in ./venv/lib/python3.10/site-packages (0.99.1) Requirement already satisfied: pydantic!=1.8,!=1.8.1,<2.0.0,>=1.7.4 in ./venv/lib/python3.10/site-packages (from fastapi==0.99.1) (1.10.12) Requirement already satisfied: starlette<0.28.0,>=0.27.0 in ./venv/lib/python3.10/site-packages (from fastapi==0.99.1) (0.27.0) Requirement already satisfied: typing-extensions>=4.5.0 in ./venv/lib/python3.10/site-packages (from fastapi==0.99.1) (4.7.1) Requirement already satisfied: anyio<5,>=3.4.0 in ./venv/lib/python3.10/site-packages (from starlette<0.28.0,>=0.27.0->fastapi==0.99.1) (3.7.1) Requirement already satisfied: exceptiongroup in ./venv/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.99.1) (1.1.2) Requirement already satisfied: idna>=2.8 in ./venv/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.99.1) (3.4) Requirement already satisfied: sniffio>=1.1 in ./venv/lib/python3.10/site-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.99.1) (1.3.0) 01:27:58-223155 INFO Version: v21.8.5 01:27:58-230558 INFO nVidia toolkit detected 01:27:59-413858 INFO Torch 2.0.1+cu118 01:27:59-418794 INFO Torch backend: nVidia CUDA 11.8 cuDNN 8700 01:27:59-429793 INFO Torch detected GPU: NVIDIA GeForce RTX 3090 VRAM 24257 Arch (8, 6) Cores 82 01:27:59-431519 INFO Verifying modules instalation status from /workspace/kohya_ss/requirements_runpod.txt... 01:27:59-435241 INFO Verifying modules instalation status from requirements.txt... 01:28:01-182956 INFO headless: True 01:28:01-186795 INFO Load CSS... Running on local URL: http://127.0.0.1:7860 Running on public URL: https://8adb4cdc7279c0b7e2.gradio.live This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces) 01:37:05-830094 INFO Copy /workspace/t to /workspace/o/img/40_soroush man... 01:37:05-857272 INFO Copy /workspace/r to /workspace/o/reg/1_man... 01:37:07-266621 INFO Done creating kohya_ss training folder structure at /workspace/o... 01:40:07-172584 INFO Start training LoRA Standard ... 01:40:07-174531 INFO Checking for duplicate image filenames in training data directory... 01:40:07-176621 INFO Valid image folder names found in: /workspace/o/img 01:40:07-178189 INFO Valid image folder names found in: /workspace/o/reg 01:40:07-179616 INFO Headless mode, skipping verification if model already exist... if model already exist it will be overwritten... 01:40:07-181335 INFO Folder 40_soroush man: 25 images found 01:40:07-182186 INFO Folder 40_soroush man: 1000 steps 01:40:07-182981 INFO [94mRegularisation images are used... Will double the number of steps required...[0m 01:40:07-183961 INFO Total steps: 1000 01:40:07-184724 INFO Train batch size: 1 01:40:07-185520 INFO Gradient accumulation steps: 1.0 01:40:07-186293 INFO Epoch: 14 01:40:07-187045 INFO Regulatization factor: 2 01:40:07-187818 INFO max_train_steps (1000 / 1 / 1.0 * 14 * 2) = 28000 01:40:07-188829 INFO stop_text_encoder_training = 0 01:40:07-189619 INFO lr_warmup_steps = 2800 01:40:07-190547 INFO Saving training config to /workspace/o/model/last_20230804-014007.json... 01:40:07-191619 INFO accelerate launch --num_cpu_threads_per_process=2 "./train_network.py" --enable_bucket --min_bucket_reso=256 --max_bucket_reso=2048 --pretrained_model_name_or_path="/workspace/Realistic_Vision_V5.0.safetensors" --train_data_dir="/workspace/o/img" --reg_data_dir="/workspace/o/reg" --resolution="512,512" --output_dir="/workspace/o/model" --logging_dir="/workspace/o/log" --network_alpha="1" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=5e-05 --unet_lr=0.0001 --network_dim=256 --output_name="last" --lr_scheduler_num_cycles="14" --no_half_vae --learning_rate="0.0001" --lr_scheduler="cosine" --lr_warmup_steps="2800" --train_batch_size="1" --max_train_steps="28000" --save_every_n_epochs="1" --mixed_precision="fp16" --save_precision="fp16" --cache_latents --optimizer_type="AdamW8bit" --max_data_loader_n_workers="0" --bucket_reso_steps=64 --bucket_no_upscale --noise_offset=0.0 2023-08-04 01:40:08.787547: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-08-04 01:40:09.446268: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT 2023-08-04 01:40:12.786000: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT prepare tokenizer Using DreamBooth method. prepare images. found directory /workspace/o/img/40_soroush man contains 25 image files No caption file found for 25 images. Training will continue without captions for these images. If class token exists, it will be used. / 25枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャプションなしで学習を続行します。class tokenが存在する場合はそれを使います。 /workspace/o/img/40_soroush man/20230711_130832.jpg /workspace/o/img/40_soroush man/20230730_100136.jpg /workspace/o/img/40_soroush man/20230730_100141.jpg /workspace/o/img/40_soroush man/20230731_153305.jpg /workspace/o/img/40_soroush man/20230731_153310.jpg /workspace/o/img/40_soroush man/20230731_153320.jpg... and 20 more found directory /workspace/o/reg/1_man contains 2734 image files No caption file found for 2734 images. Training will continue without captions for these images. If class token exists, it will be used. / 2734枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャプションなしで学習を続行します。class tokenが存在する場合はそれを使います。 /workspace/o/reg/1_man/00000-0-150-billi-P_jBxTIYGKg-unsplash.png /workspace/o/reg/1_man/00001-0-aadil-C01tGrKk11Y-unsplash.png /workspace/o/reg/1_man/00002-0-aakash-malhotra-h7mHBglEoOU-unsplash.png /workspace/o/reg/1_man/00003-0-abby-savage-PiKX8qPBFgU-unsplash.png /workspace/o/reg/1_man/00004-0-abby-savage-v8B1wpMzSfQ-unsplash.png /workspace/o/reg/1_man/00005-0-abdelrahman-ismail-OTn-RlthAQw-unsplash.png... and 2729 more 1000 train images with repeating. 2734 reg images. some of reg images are not used / 正則化画像の数が多いので、一部使用されない正則化画像があります [Dataset 0] batch_size: 1 resolution: (512, 512) enable_bucket: True min_bucket_reso: 256 max_bucket_reso: 2048 bucket_reso_steps: 64 bucket_no_upscale: True [Subset 0 of Dataset 0] image_dir: "/workspace/o/img/40_soroush man" image_count: 25 num_repeats: 40 shuffle_caption: False keep_tokens: 0 caption_dropout_rate: 0.0 caption_dropout_every_n_epoches: 0 caption_tag_dropout_rate: 0.0 color_aug: False flip_aug: False face_crop_aug_range: None random_crop: False token_warmup_min: 1, token_warmup_step: 0, is_reg: False class_tokens: soroush man caption_extension: .caption [Subset 1 of Dataset 0] image_dir: "/workspace/o/reg/1_man" image_count: 2734 num_repeats: 1 shuffle_caption: False keep_tokens: 0 caption_dropout_rate: 0.0 caption_dropout_every_n_epoches: 0 caption_tag_dropout_rate: 0.0 color_aug: False flip_aug: False face_crop_aug_range: None random_crop: False token_warmup_min: 1, token_warmup_step: 0, is_reg: True class_tokens: man caption_extension: .caption [Dataset 0] loading image sizes. 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1025/1025 [00:00<00:00, 18196.20it/s] make buckets min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is set, because bucket reso is defined by image size automatically / bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計算されるため、min_bucket_resoとmax_bucket_resoは無視されます number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む） bucket 0: resolution (512, 384), count: 1000 bucket 1: resolution (512, 512), count: 1000 mean ar error (without repeats): 0.0 preparing accelerator loading model for process 0/1 load StableDiffusion checkpoint: /workspace/Realistic_Vision_V5.0.safetensors UNet2DConditionModel: 64, 8, 768, False, False loading u-net: loading vae: loading text encoder: import network module: networks.lora [Dataset 0] caching latents. checking cache validity... 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1025/1025 [00:00<00:00, 1091434.78it/s] caching latents... 78%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 803/1025 [01:08<00:18, 11.69it/s] ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/PIL/ImageFile.py:249 in load │ │ │ │ 246 │ │ │ │ │ │ b = prefix │ │ 247 │ │ │ │ │ │ while True: │ │ 248 │ │ │ │ │ │ │ try: │ │ ❱ 249 │ │ │ │ │ │ │ │ s = read(self.decodermaxblock) │ │ 250 │ │ │ │ │ │ │ except (IndexError, struct.error) as e: │ │ 251 │ │ │ │ │ │ │ │ # truncated png/gif │ │ 252 │ │ │ │ │ │ │ │ if LOAD_TRUNCATED_IMAGES: │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/PIL/PngImagePlugin.py:929 in load_read │ │ │ │ 926 │ │ │ │ │ 927 │ │ │ self.fp.read(4) # CRC │ │ 928 │ │ │ │ │ ❱ 929 │ │ │ cid, pos, length = self.png.read() │ │ 930 │ │ │ │ │ 931 │ │ │ if cid not in [b"IDAT", b"DDAT", b"fdAT"]: │ │ 932 │ │ │ │ self.png.push(cid, pos, length) │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/PIL/PngImagePlugin.py:166 in read │ │ │ │ 163 │ │ │ s = self.fp.read(8) │ │ 164 │ │ │ cid = s[4:] │ │ 165 │ │ │ pos = self.fp.tell() │ │ ❱ 166 │ │ │ length = i32(s) │ │ 167 │ │ │ │ 168 │ │ if not is_cid(cid): │ │ 169 │ │ │ if not ImageFile.LOAD_TRUNCATED_IMAGES: │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/PIL/_binary.py:85 in i32be │ │ │ │ 82 │ │ 83 │ │ 84 def i32be(c, o=0): │ │ ❱ 85 │ return unpack_from(">I", c, o)[0] │ │ 86 │ │ 87 │ │ 88 # Output, le = little endian, be = big endian │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ error: unpack_from requires a buffer of at least 4 bytes for unpacking 4 bytes at offset 0 (actual buffer size is 0) The above exception was the direct cause of the following exception: ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /workspace/kohya_ss/./train_network.py:990 in │ │ │ │ 987 │ args = train_util.read_config_from_file(args, parser) │ │ 988 │ │ │ 989 │ trainer = NetworkTrainer() │ │ ❱ 990 │ trainer.train(args) │ │ 991 │ │ │ │ /workspace/kohya_ss/./train_network.py:252 in train │ │ │ │ 249 │ │ │ vae.requires_grad_(False) │ │ 250 │ │ │ vae.eval() │ │ 251 │ │ │ with torch.no_grad(): │ │ ❱ 252 │ │ │ │ train_dataset_group.cache_latents(vae, args.vae_batch_size, args.cache_l │ │ 253 │ │ │ vae.to("cpu") │ │ 254 │ │ │ if torch.cuda.is_available(): │ │ 255 │ │ │ │ torch.cuda.empty_cache() │ │ │ │ /workspace/kohya_ss/library/train_util.py:1824 in cache_latents │ │ │ │ 1821 │ def cache_latents(self, vae, vae_batch_size=1, cache_to_disk=False, is_main_process= │ │ 1822 │ │ for i, dataset in enumerate(self.datasets): │ │ 1823 │ │ │ print(f"[Dataset {i}]") │ │ ❱ 1824 │ │ │ dataset.cache_latents(vae, vae_batch_size, cache_to_disk, is_main_process) │ │ 1825 │ │ │ 1826 │ def cache_text_encoder_outputs( │ │ 1827 │ │ self, tokenizers, text_encoders, device, weight_dtype, cache_to_disk=False, is_m │ │ │ │ /workspace/kohya_ss/library/train_util.py:873 in cache_latents │ │ │ │ 870 │ │ # iterate batches: batch doesn't have image, image will be loaded in cache_batch │ │ 871 │ │ print("caching latents...") │ │ 872 │ │ for batch in tqdm(batches, smoothing=1, total=len(batches)): │ │ ❱ 873 │ │ │ cache_batch_latents(vae, cache_to_disk, batch, subset.flip_aug, subset.rando │ │ 874 │ │ │ 875 │ # weight_dtypeを指定するとText Encoderそのもの、およひ出力がweight_dtypeになる │ │ 876 │ # SDXLでのみ有効だが、datasetのメソッドとする必要があるので、sdxl_train_util.pyでは │ │ │ │ /workspace/kohya_ss/library/train_util.py:2135 in cache_batch_latents │ │ │ │ 2132 │ """ │ │ 2133 │ images = [] │ │ 2134 │ for info in image_infos: │ │ ❱ 2135 │ │ image = load_image(info.absolute_path) if info.image is None else np.array(info. │ │ 2136 │ │ # TODO 画像のメタデータが壊れていて、メタデータから割り当てたbucketと実際の画像 │ │ 2137 │ │ image, original_size, crop_ltrb = trim_and_resize_if_required(random_crop, image │ │ 2138 │ │ image = IMAGE_TRANSFORMS(image) │ │ │ │ /workspace/kohya_ss/library/train_util.py:2084 in load_image │ │ │ │ 2081 │ image = Image.open(image_path) │ │ 2082 │ if not image.mode == "RGB": │ │ 2083 │ │ image = image.convert("RGB") │ │ ❱ 2084 │ img = np.array(image, np.uint8) │ │ 2085 │ return img │ │ 2086 │ │ 2087 │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/PIL/Image.py:673 in __array_interface__ │ │ │ │ 670 │ │ │ │ # See: https://github.com/python-pillow/Pillow/issues/350 │ │ 671 │ │ │ │ new["data"] = self.tobytes("raw", "L") │ │ 672 │ │ │ else: │ │ ❱ 673 │ │ │ │ new["data"] = self.tobytes() │ │ 674 │ │ except Exception as e: │ │ 675 │ │ │ if not isinstance(e, (MemoryError, RecursionError)): │ │ 676 │ │ │ │ try: │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/PIL/Image.py:732 in tobytes │ │ │ │ 729 │ │ if encoder_name == "raw" and args == (): │ │ 730 │ │ │ args = self.mode │ │ 731 │ │ │ │ ❱ 732 │ │ self.load() │ │ 733 │ │ │ │ 734 │ │ if self.width == 0 or self.height == 0: │ │ 735 │ │ │ return b"" │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/PIL/ImageFile.py:256 in load │ │ │ │ 253 │ │ │ │ │ │ │ │ │ break │ │ 254 │ │ │ │ │ │ │ │ else: │ │ 255 │ │ │ │ │ │ │ │ │ msg = "image file is truncated" │ │ ❱ 256 │ │ │ │ │ │ │ │ │ raise OSError(msg) from e │ │ 257 │ │ │ │ │ │ │ │ │ 258 │ │ │ │ │ │ │ if not s: # truncated jpeg │ │ 259 │ │ │ │ │ │ │ │ if LOAD_TRUNCATED_IMAGES: │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ OSError: image file is truncated ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /workspace/kohya_ss/venv/bin/accelerate:8 in │ │ │ │ 5 from accelerate.commands.accelerate_cli import main │ │ 6 if __name__ == '__main__': │ │ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │ │ ❱ 8 │ sys.exit(main()) │ │ 9 │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py:45 │ │ in main │ │ │ │ 42 │ │ exit(1) │ │ 43 │ │ │ 44 │ # Run │ │ ❱ 45 │ args.func(args) │ │ 46 │ │ 47 │ │ 48 if __name__ == "__main__": │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py:918 in │ │ launch_command │ │ │ │ 915 │ elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA │ │ 916 │ │ sagemaker_launcher(defaults, args) │ │ 917 │ else: │ │ ❱ 918 │ │ simple_launcher(args) │ │ 919 │ │ 920 │ │ 921 def main(): │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py:580 in │ │ simple_launcher │ │ │ │ 577 │ process.wait() │ │ 578 │ if process.returncode != 0: │ │ 579 │ │ if not args.quiet: │ │ ❱ 580 │ │ │ raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) │ │ 581 │ │ else: │ │ 582 │ │ │ sys.exit(1) │ │ 583 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python', './train_network.py', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--pretrained_model_name_or_path=/workspace/Realistic_Vision_V5.0.safetensors', '--train_data_dir=/workspace/o/img', '--reg_data_dir=/workspace/o/reg', '--resolution=512,512', '--output_dir=/workspace/o/model', '--logging_dir=/workspace/o/log', '--network_alpha=1', '--save_model_as=safetensors', '--network_module=networks.lora', '--text_encoder_lr=5e-05', '--unet_lr=0.0001', '--network_dim=256', '--output_name=last', '--lr_scheduler_num_cycles=14', '--no_half_vae', '--learning_rate=0.0001', '--lr_scheduler=cosine', '--lr_warmup_steps=2800', '--train_batch_size=1', '--max_train_steps=28000', '--save_every_n_epochs=1', '--mixed_precision=fp16', '--save_precision=fp16', '--cache_latents', '--optimizer_type=AdamW8bit', '--max_data_loader_n_workers=0', '--bucket_reso_steps=64', '--bucket_no_upscale', '--noise_offset=0.0']' returned non-zero exit status 1.

Virtamouse

Running into this error. CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python', './sdxl_train_network.py', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--pretrained_model_name_or_path=/workspace/sd_xl_base_1.0.safetensors', '--train_data_dir=/workspace/koya files/img', '--reg_data_dir=/workspace/koya files/reg', '--resolution=1024,1024', '--output_dir=/workspace/koya files/model', '--logging_dir=/workspace/koya files/log', '--network_alpha=1', '--save_model_as=safetensors', '--network_module=networks.lora', '--text_encoder_lr=4e-05', '--unet_lr=0.0004', '--network_dim=256', '--output_name=cebgtt', '--lr_scheduler_num_cycles=10', '--no_half_vae', '--learning_rate=0.0004', '--lr_scheduler=constant', '--train_batch_size=1', '--max_train_steps=7600', '--save_every_n_epochs=1', '--mixed_precision=bf16', '--save_precision=bf16', '--cache_latents', '--cache_latents_to_disk', '--optimizer_type=Adafactor', '--optimizer_args', 'scale_parameter=False', 'relative_step=False', 'warmup_init=False', '--max_data_loader_n_workers=0', '--bucket_reso_steps=64', '--gradient_checkpointing', '--xformers', '--bucket_no_upscale', '--noise_offset=0.0']' died with .

Virtamouse

/workspace/kohya_ss/venv/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:339: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() and inp.query.storage().data_ptr() == inp.key.storage().data_ptr() steps: 10%|█████████▏ | 760/7600 [11:56<1:47:26, 1.06it/s, loss=0.113] saving checkpoint: /workspace/koya files/model/cebgtt-000001.safetensors ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /workspace/kohya_ss/venv/bin/accelerate:8 in │ │ │ │ 5 from accelerate.commands.accelerate_cli import main │ │ 6 if __name__ == '__main__': │ │ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │ │ ❱ 8 │ sys.exit(main()) │ │ 9 │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py:45 │ │ in main │ │ │ │ 42 │ │ exit(1) │ │ 43 │ │ │ 44 │ # Run │ │ ❱ 45 │ args.func(args) │ │ 46 │ │ 47 │ │ 48 if __name__ == "__main__": │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py:918 in │ │ launch_command │ │ │ │ 915 │ elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA │ │ 916 │ │ sagemaker_launcher(defaults, args) │ │ 917 │ else: │ │ ❱ 918 │ │ simple_launcher(args) │ │ 919 │ │ 920 │ │ 921 def main(): │ │ │ │ /workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py:580 in │ │ simple_launcher │ │ │ │ 577 │ process.wait() │ │ 578 │ if process.returncode != 0: │ │ 579 │ │ if not args.quiet: │ │ ❱ 580 │ │ │ raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) │ │ 581 │ │ else: │ │ 582 │ │ │ sys.exit(1) │ │ 583 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python', './sdxl_train_network.py', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--pretrained_model_name_or_path=/workspace/sd_xl_base_1.0.safetensors', '--train_data_dir=/workspace/koya files/img', '--reg_data_dir=/workspace/koya files/reg', '--resolution=1024,1024', '--output_dir=/workspace/koya files/model', '--logging_dir=/workspace/koya files/log', '--network_alpha=1', '--save_model_as=safetensors', '--network_module=networks.lora', '--text_encoder_lr=4e-05', '--unet_lr=0.0004', '--network_dim=256', '--output_name=cebgtt', '--lr_scheduler_num_cycles=10', '--no_half_vae', '--learning_rate=0.0004', '--lr_scheduler=constant', '--train_batch_size=1', '--max_train_steps=7600', '--save_every_n_epochs=1', '--mixed_precision=bf16', '--save_precision=bf16', '--cache_latents', '--cache_latents_to_disk', '--optimizer_type=Adafactor', '--optimizer_args', 'scale_parameter=False', 'relative_step=False', 'warmup_init=False', '--max_data_loader_n_workers=0', '--bucket_reso_steps=64', '--gradient_checkpointing', '--xformers', '--bucket_no_upscale', '--noise_offset=0.0']' died with .

erwan parso

I have successfully installed the system, but when I start my first training I get an error message : 17:44:16-392240 INFO accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train_network.py" --enable_bucket --min_bucket_reso=256 --max_bucket_reso=2048 --pretrained_model_name_or_path="/workspace/stable-diffusion-webui/models/Stable-diffusion/sd_xl_base_1.0.safetensors" --train_data_dir="/workspace/train/melissa/img" --reg_data_dir="/workspace/train/melissa/reg" --resolution="1024,1024" --output_dir="/workspace/train/melissa/model" --logging_dir="/workspace/train/melissa/log" --network_alpha="1" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --network_dim=256 --output_name="melsdxl" --lr_scheduler_num_cycles="10" --no_half_vae --learning_rate="0.0004" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="13600" --save_every_n_epochs="1" --mixed_precision="bf16" --save_precision="bf16" --caption_extension="txt" --cache_latents --cache_latents_to_disk --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers="0" --bucket_reso_steps=64 --gradient_checkpointing --bucket_no_upscale --noise_offset=0.0 Illegal instruction (core dumped)

Ran Bensimon

Hey, thanks for the great resouece! I'm getting an error when trying to train: OutOfMemoryError: CUDA out of memory. using runpod: RTX 4090 12 vCPU 54 GB RAM runpod/stable-diffusion:web-ui-10.0.0

Ran Bensimon

Wow thanks for the quick reply!! I'm making sure now to have all the parametes like in your tut. I was using another config file. so I'll double check and update ASAP.

Ran Bensimon

unfortunately still getting the error, tried with and withou xformers

Ran Bensimon

3090 is working as expected! :)

Ran Bensimon

Is there a way to work with more than 1 GPU? I'm using 2 x RTX 3090 but only one is active.

SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts

i never tried combining power of both for training. but you can do seperate training on each one of them it is easy. add this command to the gui.bat file to make it use second GPU set CUDA_VISIBLE_DEVICES=1 I may work on dual GPU tutorial hopefully soon

Ran Bensimon

I understand, you mean using two separate pods, right?

SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts

no in a single pod you can also use. as long as your pod has more than 1 gpu you can do training on each one or combine their power. if you combine their power it would be like 1 image processed on 1 gpu another one processed on second gpu. but this thing i never tested yet.

Zlata Ponirovskaya

Dear Dr. Furkan Gözükara could you please help me via Zoom/Discord/TeamViewer session? I'll pay for your time. Thank you in advance! (my discord 879480280983961681 possuminside / whatsapp +447751674250 )

Gerenier

This is very helpful to use Kohya with runpod. Great work! I only had a minor problem. Please have a look at relauncher.py. It seems that it will not prevent stable-diffusion-webui from launch after the restart of the pod.

Rafał Ryniak

Hey, do you have koyass presets for SD1.5 and SDXL for training, I'm so lost with the settings that my head will explode, I would be very grateful.

Rafał Ryniak

sorry, I missed it, the answer was very fast anyway, thank you very much, the last question, if I train lore with this preset, will it work with checkpoints 1.5

Rafał Ryniak

Hi, where is it better to train lora, koya_ss, or in the controlnet dreambooth extension?, (model realisticvision 5), thanks and best regards.

Rafał Ryniak

Hi, are the model files for lora KoyaSS training (SDXL can be 1024x1536 or better 1024x1024) and for training (SD 1.5 512x768 or 512x512 and should the size of reference files be the same size as the size of the model, or does it matter? Regards.

Rafał Ryniak

13:31:15-925441 INFO Save... 13:31:21-340913 INFO Start training LoRA Standard ... 13:31:21-341826 INFO Checking for duplicate image filenames in training data directory... 13:31:21-342498 INFO Valid image folder names found in: /workspace/stable-diffusion-webui/models/Lora/img 13:31:21-343023 INFO Valid image folder names found in: /workspace/stable-diffusion-webui/models/Lora/reg 13:31:21-343513 INFO Headless mode, skipping verification if model already exist... if model already exist it will be overwritten... 13:31:21-344095 INFO Folder 25_terk woman: 17 images found 13:31:21-344565 INFO Folder 25_terk woman: 425 steps 13:31:21-345020 WARNING Regularisation images are used... Will double the number of steps required... 13:31:21-345520 INFO Total steps: 425 13:31:21-345964 INFO Train batch size: 1 13:31:21-346403 INFO Gradient accumulation steps: 1 13:31:21-346860 INFO Epoch: 8 13:31:21-347285 INFO Regulatization factor: 2 13:31:21-347728 INFO max_train_steps (425 / 1 / 1 * 8 * 2) = 6800 13:31:21-348303 INFO stop_text_encoder_training = 0 13:31:21-348750 INFO lr_warmup_steps = 0 13:31:21-349242 INFO Saving training config to /workspace/stable-diffusion-webui/models/Lora/model/terk_20230823-133121.json... 13:31:21-349974 INFO accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train_network.py" --pretrained_model_name_or_path="/workspace/stable-diffusion-webui/models/Stable-diffusion/sd_xl_base_1.0.safetensors" --train_data_dir="/workspace/stable-diffusion-webui/models/Lora/img" --reg_data_dir="/workspace/stable-diffusion-webui/models/Lora/reg" --resolution="1024,1024" --output_dir="/workspace/stable-diffusion-webui/models/Lora/model" --logging_dir="/workspace/stable-diffusion-webui/models/Lora/log" --network_alpha="1" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --network_dim=32 --output_name="terk" --lr_scheduler_num_cycles="8" --no_half_vae --learning_rate="0.0004" --lr_scheduler="constant" --train_batch_size="1" --max_train_steps="6800" --save_every_n_epochs="1" --mixed_precision="bf16" --save_precision="bf16" --cache_latents --cache_latents_to_disk --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers="0" --bucket_reso_steps=64 --gradient_checkpointing --xformers --bucket_no_upscale --noise_offset=0.0 Illegal instruction (core dumped)

Rafał Ryniak

3090 ID: b1de4n1gohnbwc 1 x RTX 3090 9 vCPU 37 GB RAM runpod/stable-diffusion:web-ui-10.1.0 On-Demand - Community Cloud Running 10 GB Disk 120 GB Pod Volume Volume Path: /workspace US 891 Mbps 878 Mbps 1511 MBps Pod Uptime: 21m 31 seconds ago Pod Utilization CPU 0% Mem 13% Disk Utilization Container 1% Volume 43% GPU Utilization 0 0% GPU Memory Used 0 12%

Rafał Ryniak

Thanks, it actually started on another capsule

noob gnosis

Hey thanks so much for the tutorial, glad i subscribed. Am i supposed to use the sd_xl_base_1.0.safetensors that comes with the .json config or download sdxl_vae.safetensors as the model for lora training? also, in your video you say to turn off xformers, but the .json you have in this post uses xformers. Which is better at this time?

Doc Snyder

Copying accelerate config file to: /root/.cache/huggingface/accelerate/default_config.yaml Traceback (most recent call last): File "/workspace/kohya_ss/setup/validate_requirements.py", line 19, in from library.custom_logging import setup_logging File "/workspace/kohya_ss/library/custom_logging.py", line 6, in from rich.theme import Theme ModuleNotFoundError: No module named 'rich' And "pip install rich" didn't fix the problem.

Doc Snyder

RunPod Stable Diffusion runpod/stable-diffusion:web-ui-10.2.1 Tryed with 25GB Contianer size..same error

Doc Snyder

while running kohya_installer.sh

Doc Snyder

Ok sorry my fault, I tryed "sh kohya_installer.sh"... now it works

mike oxmaul

When creating kohya dataset via tools > prepare training data "Traceback (most recent call last): File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/routes.py", line 439, in run_predict output = await app.get_blocks().process_api( File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1384, in process_api result = await self.call_function( File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1089, in call_function prediction = await anyio.to_thread.run_sync( File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2106, in run_sync_in_worker_thread return await future File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 833, in run result = context.run(func, *args) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/utils.py", line 700, in wrapper response = f(*args, **kwargs) File "/workspace/kohya_ss/library/dreambooth_folder_creation_gui.py", line 52, in dreambooth_folder_preparation msgbox('Class prompt missing...') File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/easygui/boxes/derived_boxes.py", line 230, in msgbox return buttonbox(msg=msg, File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/easygui/boxes/button_box.py", line 95, in buttonbox bb = ButtonBox( File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/easygui/boxes/button_box.py", line 147, in __init__ self.ui = GUItk(msg, title, choices, images, default_choice, cancel_choice, self.callback_ui) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/easygui/boxes/button_box.py", line 263, in __init__ self.boxRoot = tk.Tk() File "/usr/lib/python3.10/tkinter/__init__.py", line 2299, in __init__ self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use) _tkinter.TclError: no display name and no $DISPLAY environment variable"

mike oxmaul

The webui seems quite different since the latest update to kohya.

WMK

In this video (https://www.youtube.com/watch?v=-xEwaQ54DI4) you mention "Automatic Installation" vs the "Manual Installation" and I'm having trouble understanding the purpose of each and why those names are given to each method (aside from the fact that one uses a script). However, I realize this page seems dedicated to "automatic" installation. My plan is to use Kohya to train LORA models. I plan to use a Network Volume that I created on RunPod so that my filesystem is persistent in case a pod becomes unavailable (in which case, I won't have to reconfigure the filesystem). With network drives on RunPod you can connect to Pods on the Secure Cloud that maintain your file system. I am using the Fast Stable Diffusion template for my Pod. I am not sure if that configuration changes the answer to my question, so I just wanted to mention it. After watching the video tutorial, and reading through the instructions above I executed the script below to perform the installation: kohya_installer.sh I then executed the following commands: apt update yes | apt-get install python3.10-tk fuser -k 7860/tcp cd /workspace/kohya_ss source venv/bin/activate pip install fastapi==0.99.1 bash gui.sh --share --headless which provided the gradio link to launch the Kohya GUI, although I still remain confused about the other Installation method you mentioned in the video as the “Manual installation." Under what circumstances should that be used?

SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts

1st : manual installation is shown for people who are not patreon subscriber nothing else 2nd : your copy pasted command is incorrect apt update yes | apt-get install python3.10-tk fuser -k 7860/tcp cd /workspace/kohya_ss source venv/bin/activate pip install fastapi==0.99.1 bash gui.sh --share --headless

WMK

Thanks for the fast reply and clarification. I noticed right after I posted my original comment that the copy/paste command ruined the -- for the input arguments. Is that what you are referring to?

Goldwaters

Does this line also apply for GPUs other than A5000? "apt install libcudnn8=8.9.5.29-1+cuda11.8 libcudnn8-dev=8.9.5.29-1+cuda11.8" I'd like to test with higher batch sizes so looking at running with higher VRAM runpods

Chris

is the thing still broken with new version of kohya? I've just trained a lora on runpod with the above instructions and the face is terrible she looks very overweight despite being slim in all the pictures used. Not sure what the issue is.

Jean Rocher

Hey, The code stopped running after 8 hours, halfway through steps 46%... GPU is still active. Any idea what's the matter? Maybe I can get it back without running all over again?

Jean Rocher

Yea, i'm talking about Jupyter commands, not the webui. The Jupyter command stopped at 46% and just got stuck

Nerdlicht

I get a warning when installing Kohya: 21:37:13-082744 WARNING Could not automatically configure accelerate. Please manually configure accelerate with the option in the menu or with: accelerate config. Copying accelerate config file to: /root/.cache/huggingface/accelerate/default_config.yaml Can I ignore that?

Felix Rockwell

Hello, do I need to reran all the scripts if I stop and start runpod?

SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts

only this part if you are using same pod also if it is web ui pod dont forget fuser -k 3000/tcp to kill web ui apt update yes | apt-get install python3.10-tk fuser -k 7860/tcp cd /workspace/kohya_ss source venv/bin/activate bash gui.sh --share --headless

Felix Rockwell

I followed this tutorial but I got this error message after a few hours of training: safetensors_rust.SafetensorError: Error while serializing: IoError(Os { code: 28, kind: StorageFull, message: "No space left on device" }) So it's not possible to train my model on runpod if there is not enough storage?

Felix Rockwell

Hello, I downloaded the lora files to huggingface and then I wanted to download them again the next day, but if I download them with wget to the correct folder, the lora files don't appear on Stable diffusion, it shows no lora files, do you know please why?

Felix Rockwell

If I am installing Kohya on Runpod with 24GB_TextEncoder.json - Parameters - Advanced there is VAE with path - stabilityai/sdxl-vae. Is that correct? It seems like a wrong path for Runpod.

grzegorz sokołowski

can i use other models than base one? i got errors when i tried to use model from civitai

Nik Dhabarde

Getting --train_text_encoder issue when using 24GB_TextEncoder.json on Runpod, any idea what might be causing this?

Nenad Kuzmanovic

Kohya had massive update. This script installs latest version, or particular?

Nenad Kuzmanovic

Can i just replace hash in your installer? I will try that

design master

runpod kohya can no longer run, please fix it, thank you, I received the following error message: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behavior is the source of the following dependency conflicts. gradio 4.19.2 requires pydantic>=2.0, but you have pydantic 1.10.15 which is incompatible.

Sebastian Grönvall

Hey I cant seem to download the files, am I doing something wrong?

Sebastian Grönvall

any files, it says that I dont have permission on computer, seems like I can view the files on phone tho

Ася Александрова

Hello sir, are there any updates for the manual Kohya installation according to this kind of error? I can't use your automatic installer since I'm consulting my colleagues and can't share your file among them all. Thanks!

Ася Александрова

I just tried the automatic installer but it ended up with the same error: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. gradio 4.26.0 requires pydantic>=2.0, but you have pydantic 1.10.15 which is incompatible. ImportError: cannot import name 'RootModel' from 'pydantic' (/workspace/kohya_ss/venv/lib/python3.10/site-packages/pydantic/__init__.cpython-310-x86_64-linux-gnu.so)

Diggy Dre

I know this seems crazy but I wanted to go ahead and ask... What kohya settings would be okay to train an SDXL LoRA with 500 images of a person? It is for a LoRA that I plan to use frequently, as the person will become a dynamic character. I have almost thought of breaking the dataset into smaller categories and training those as separate LoRA models, which may be a good option as long as I can keep the person/character looking as it is intended to look. I have retrained it about 6 times already using a network rank of 64 (for some reason, 128 looks horrible) and an alpha of 1. I have adjusted the learning rates. On a few of them, I feel like I am ALMOST there, as if maybe I was only off by an epoch or two. I have been following you for a while and everything that you have taught me so far is of great value, so I figured it would be worth it to ask you.

SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts

my research is focused on dreambooth / fine tuning. because it is far better than LoRA training and it is proven by a research paper. just train with my config, then extract LoRA and you will get much better quality. it is whether 500 images or not doesnt matter. only the number of epochs / steps you need changes. so try this. 500 images, 10 repeat, 10 epoch and save every 1 epoch to see how it goes.

Diggy Dre

I will give it a shot. Thank you so much!

Ася Александрова

I did this: 'apt update yes | apt-get install python3.10-tk' on a new pod and then tried to install the automatic kohya installer and it ended with an error I haven't seen before (screenshot: https://i.ibb.co/WP3d9Ky/photo-2024-04-28-20-51-10.jpg). Although previously, before this update, at least the first step of installation worked fine. My attempt to run the second step of installation ended up with a 'pydantic' error again

Diggy Dre

What method/settings are you using to extract the LoRA? I made an awesome checkpoint using your advice but when I try to extract a LoRA from it using Kohya, I actually am losing the likeness of the person that was trained.

Diggy Dre

Can I train a checkpoint using a base model that has a baked VAE or would that be bad?

Diggy Dre

Awesome! Thank you. I trained my checkpoint just the way that you told me to, but I actually left off the regularization images. I think it looks better than a LoRA but it doesn't really pull in the skin texture as well and I think it could actually be better. I am thinking to use something like RealVisXL and do a retrain.

Diggy Dre

If I go for using the regularization images this time, do I still set everything for 10 repeats?

Diggy Dre

Awesome! Thank you so much!

BridgeWater

Greetings dear Furkan, I have 2 remarks regarding your great walkthrough on the Authomatic Kohya SS Lora 1. If I may say it, You have to update your video tutorials about Stable Diffusion XL (SDXL) DreamBooth Training, or make a new one, the GUI is not the same anymore, and looks differently. Moreover, the Train/ Preprocess images is not there anymore, meaning that the user has to crop/resize the images manually. By the way, do You have Any tips on resizing for the community? 2. On your last tutorial for Patreon subscribers (here : https://www.youtube.com/watch?v=EEV8RPohsbw) You advice to put the Pretrained model name or path as “SD_XL_base_1.0.safetensors” this leads now to the “Size mismatch for model.diffusion_model.output_blocks.8.0.skip_connection.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([640])” error. The safest way to do it is to put “v1-5-pruned.ckpt” as model. I can see that You did put a screenshot advising to choose between one of the 2. But the former doesn’t work anymore. Again, if you have an observation, or remark for why it doesn’t work anymore, feel free to share it.

SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts

thanks for suggestions. I am totally agree that GUI got a lot of updates thus I have to make a new video for it. Also for cropping I suggest you this video for best cropping : https://youtu.be/Fbuyu35TkE4 about model naming i suggest you to download model into wherever you are running first and give full path. because when gui uses hugging face repo model sometimes working sometimes not. best is giving local model full path

BridgeWater

Hello again dear Furkan, It has been 2 days, and endless tries to get to the result that the instructions of this post don't work anymore on Runpod. I repeated carefully step after step the instructions. Tried following the video made for patreons here: https://www.youtube.com/watch?v=EEV8RPohsbw . But nothing. Where does it fail? Let’s get back to the instructions: • Starting a stable-diffusion:web-ui in Runpod ✅ • Make container disk size at least 15 GB – Made 20 Go ✅ • Download attached kohya_installer.sh, downloadSDXL.sh and then uploading them into the Workspace folder ✅ • Reload the pod after overwriting downloadSDXL.sh ✅ • Using the quick command : chmod +x kohya_installer.sh ./kohya_installer.sh ✅ • Creating a folder with the “Train” pictures ✅ • Creating a folder with the “Regulation” pictures ✅ • Killing the previous instance before running Kohya with the command : fuser -k 3000/tcp ✅ • The GUI connected and running ✅ It is now that it doesn’t work anymore. I’ve tried both the video advices and the configuration screen shot on the GUI (runpod_8400_Steps.png , by the way some tabs are not developed and would be better if all are seen), but it doesn’t work… What are the errors ? If I put the model as “Stable-diffusion/v1-5-pruned.ckpt” The error is : --------------------------------------------- 2024-08-12 00:11:17 INFO use Adafactor optimizer | {'relative_step': True} train_util.py:4485 INFO relative_step is true / relative_stepがtrueです train_util.py:4488 WARNING learning rate is used as initial_lr / 指定したlearning rateはinitial_lrとして使用されます train_util.py:4490 WARNING unet_lr and text_encoder_lr are ignored / unet_lrとtext_encoder_lrは無視されます train_util.py:4502 INFO use adafactor_scheduler / スケジューラにadafactor_schedulerを使用します train_util.py:4507 Traceback (most recent call last): File "/workspace/kohya_ss/sd-scripts/train_db.py", line 529, in train(args) File "/workspace/kohya_ss/sd-scripts/train_db.py", line 215, in train lr_scheduler = train_util.get_scheduler_fix(args, optimizer, accelerator.num_processes) File "/workspace/kohya_ss/sd-scripts/library/train_util.py", line 4595, in get_scheduler_fix return wrap_check_needless_num_warmup_steps(transformers.optimization.AdafactorSchedule(optimizer, initial_lr)) File "/workspace/kohya_ss/sd-scripts/library/train_util.py", line 4572, in wrap_check_needless_num_warmup_steps raise ValueError(f"{name} does not require `num_warmup_steps`. Set None or 0.") ValueError: adafactor:1e-05 does not require `num_warmup_steps`. Set None or 0. Traceback (most recent call last): File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in sys.exit(main()) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1017, in launch_command simple_launcher(args) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 637, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python', '/workspace/kohya_ss/sd-scripts/train_db.py', '--config_file', 'workspace/stable-diffusion-webui/models/Stable-diffusion/model/config_dreambooth-20240812-015057.toml']' returned non-zero exit status 1. 00:11:19-447697 INFO Training has ended. --------------------------------------------- If I put the model as “sd_xl_base_1.0.safetensors” --------------------------------------------- Traceback (most recent call last): File "/workspace/kohya_ss/sd-scripts/train_db.py", line 529, in train(args) File "/workspace/kohya_ss/sd-scripts/train_db.py", line 123, in train text_encoder, vae, unet, load_stable_diffusion_format = train_util.load_target_model(args, weight_dtype, accelerator) File "/workspace/kohya_ss/sd-scripts/library/train_util.py", line 4825, in load_target_model text_encoder, vae, unet, load_stable_diffusion_format = _load_target_model( File "/workspace/kohya_ss/sd-scripts/library/train_util.py", line 4780, in _load_target_model text_encoder, vae, unet = model_util.load_models_from_stable_diffusion_checkpoint( File "/workspace/kohya_ss/sd-scripts/library/model_util.py", line 1008, in load_models_from_stable_diffusion_checkpoint info = unet.load_state_dict(converted_unet_checkpoint) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2152, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel: Missing key(s) in state_dict: "down_blocks.0.attentions.0.norm.weight", "down_blocks.0.attentions.0.norm.bias", "down_blocks.0.attentions.0.proj_in.weight", "down_blocks.0.attentions.0.proj_in.bias", "down_blocks.0.attentions.0.transformer_blocks.0.attn1.to_q.weight", "down_blocks.0.attentions.0.transformer_blocks.0.attn1.to_k.weight", "down_blocks.0.attentions.0.transformer_blocks.0.attn1.to_v.weight", "down_blocks.0.attentions.0.transformer_blocks.0.attn1.to_out.0.weight", "down_blocks.0.attentions.0.transformer_blocks.0.attn1.to_out.0.bias", "down_blocks.0.attentions.0.transformer_blocks.0.ff.net.0.proj.weight", "down_blocks.0.attentions.0.transformer_blocks.0.ff.net.0.proj.bias", "down_blocks.0.attentions.0.transformer_blocks.0.ff.net.2.weight", "down_blocks.0.attentions.0.transformer_blocks.0.ff.net.2.bias", "down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_q.weight", "down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_k.weight", "down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_v.weight", "down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_out.0.weight", "down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_out.0.bias", "down_blocks.0.attentions.0.transformer_blocks.0.norm1.weight", "down_blocks.0.attentions.0.transformer_blocks.0.norm1.bias", "down_blocks.0.attentions.0.transformer_blocks.0.norm2.weight", "down_blocks.0.attentions.0.transformer_blocks.0.norm2.bias", "down_blocks.0.attentions.0.transformer_blocks.0.norm3.weight", "down_blocks.0.attentions.0.transformer_blocks.0.norm3.bias", "down_blocks.0.attentions.0.proj_out.weight", "down_blocks.0.attentions.0.proj_out.bias", "down_blocks.0.attentions.1.norm.weight" (...) size mismatch for down_blocks.1.attentions.0.proj_in.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([640, 640, 1, 1]). size mismatch for down_blocks.1.attentions.0.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is torch.Size([640, 768]). size mismatch for down_blocks.1.attentions.0.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([640, 2048]) from checkpoint, the shape in current model is torch.Size([640, 768]). size mismatch for down_blocks.1.attentions.0.proj_out.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([640, 640, 1, 1]). size mismatch for down_blocks.1.attentions.1.proj_in.weight: copying a param with shape torch.Size([640, 640]) from checkpoint, the shape in current model is torch.Size([640, 640, 1, 1]). (...) File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in sys.exit(main()) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1017, in launch_command simple_launcher(args) File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 637, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python', '/workspace/kohya_ss/sd-scripts/train_db.py', '--config_file', 'workspace/stable-diffusion-webui/models/Stable-diffusion/model/config_dreambooth-20240812-021311.toml']' returned non-zero exit status 1. 00:13:20-950565 INFO Training has ended. ------------------------------------------ Any help ? Or Shall I abandon these tutorials because they are outdated ?

BridgeWater

Thanks for your reply ! For the first message: No, I was using the full path in the Stable diffusion folder. For the second message : How did You update it ? Is it now the good version if someone follows the instructions from the beginning in the current post we are commenting ? OK, I saw the updates on the 2 posts, but: 1. the screen shot in the GUI of the "The Very Best Kohya GUI Workflow & Config For SD 1.5 Based Models DreamBooth / Full Fine Tuning" does not include any of the tips in Your videos (e.g. bf16 selected , Repeats number, max resolution 1024x1024 ...) 2. does that mean that the current post we are commenting is outdated ? Again thanks!

Deniz Oliveira Erdinc

Dear Furkan, I get this error when trying to run kohya on runpod. I did everything as you said on the tutorial up to here but it doesnt work. Traceback (most recent call last): File "/workspace/kohya_ss/kohya_gui.py", line 6, in import gradio as gr File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/__init__.py", line 3, in import gradio._simple_templates File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/_simple_templates/__init__.py", line 1, in from .simpledropdown import SimpleDropdown File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/_simple_templates/simpledropdown.py", line 7, in from gradio.components.base import Component, FormComponent File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/components/__init__.py", line 1, in from gradio.components.annotated_image import AnnotatedImage File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/components/annotated_image.py", line 14, in from gradio import processing_utils, utils File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/processing_utils.py", line 29, in from gradio import utils, wasm_utils File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/utils.py", line 67, in from gradio.data_classes import ( File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/gradio/data_classes.py", line 25, in from pydantic import ( ImportError: cannot import name 'GetCoreSchemaHandler' from 'pydantic' (/workspace/kohya_ss/venv/lib/python3.10/site-packages/pydantic/__init__.cpython-310-x86_64-linux-gnu.so)

Ariana Coral

how I can extract the LoRa of my dreambooth sdxl?

Heinrich Viljoen

Hi Furkan, I am trying to use the auto installer, but when I use the second command the following happens: root@5f7ab06a67a1:/workspace# chmod +x kohya_installer.sh ./kohya_installer.sh chmod: cannot access 'kohya_installer.sh': No such file or directory

Digital Coach

First of all Thank for you all of your sharing. i have followed your steps to train SDXL LoRA and after training when i use this LoRA in comfyui its generating cartoonic Images not realistic Human. i followed same ass your steps.

Digital Coach

i am using 45 of my Real Images and giving random prompts but results are same. if you can provide me setting of Kohya JSon file then i use it and test that what's wrong with it. i am using Runpod and also train on my local PC with RTX3060 12GB but results are same.