Home
Creators
Search Recent Random
Posts
Search DMs Popular Hash Lookup Tags Random
Importer
Import FAQ
Register Favorites Keys Review DMs Logout
Community
Telegram ThePornDude
Home Creators Posts Import Register Favorites Logout

JoyCaption : Amazing Image Captioner - Batch Processing and Multi-GPU - Windows, RunPod, Massed Compute 1-Click Installers (Patreon)

Published:
2023-11-27 22:11:00
Edited:
2024-12-28 14:41:00
Imported:
2026-06

Downloads

Content

This post is now obsolete : https://www.patreon.com/posts/118827960

New post : https://www.patreon.com/posts/118827960

New scripts are published at : https://www.patreon.com/posts/118827960

https://www.patreon.com/posts/118827960

Patreon exclusive posts index to find our scripts easily, Patreon scripts updates history to see which updates arrived to which scripts and amazing Patreon special generative scripts list that you can use in any of your task.

Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord

Please also Star, Watch and Fork our Stable Diffusion & Generative AI  GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)

=======

For mass editing captions > https://www.patreon.com/posts/108992085

Latest installers : Joy_Caption_v25.zip

24 November 2024 Update V23

Stop batch processing button added as requested

24 November 2024 Update V23

Forced resize to 384px removed and now clip_processor will process and resize images

Batch size fixed for newer Alpha One app but i felt like it slowed down for RTX 3090 on 4-bit mode

Now models will be downloaded way faster

22 September 2024 Update V20

Joy Caption Alpha One added to the apps

Now you can start both older Pre-Alpha and Alpha One via Windows start

Cloud instructions also have details to start older or newer version

4-bit mode works perfect and uses as low as 8.5 GB GPU with newer Alpha One

Make a fresh install into a new folder please

After very first run (wait till download models and generate caption), restart app to make it fully accurately working

We started using original LLAMA therefore you should delete older one that is in your Hugging Face cache to empty space

Tutorial for this : https://www.patreon.com/posts/108419878

The folder that you need to delete is models--rombodawg--Meta-Llama-3.1-8B-Instruct-reuploaded (optional but you save space)

Please upvote this thread if possible : https://www.reddit.com/r/FluxAI/comments/1fmupnm/detailed_comparison_of_joycaption_alpha_one_vs/

19 September 2024 Update V17

Now it will display as JoyCaption in the browser tab

A new feature remove repeating (duplicate) sentences added and it is enabled by default

15 September 2024 Update V16

Optional prefix and suffix added to the app (don't forget to add space char to beginning or end)

Optional don't save captioned images checkbox added to the app

Just overwrite previous app.py file for updating

Torch version upgraded to 2.4.1

27 August 2024 Update V12

A bug which was causing saved images in outputs folders to be resized fixed - now original input images saved

Batch size processing added - can increase your processing speed

26 August 2024 Update V11

Massive updates arrived

Now fully supports multi GPU training

Tested on 8X A6000 on Massed Compute and RTX 3090 TI + RTX 3060 on my windows PC

New Resize Input Image Max Resolution added - i find 1536 works best

Overwrite previous files for update

Set multi GPU ids as 0,1,2,3 etc how many you have

Full example images posted here : https://www.reddit.com/r/FluxAI/comments/1f1eaf3/afaik_first_fully_multigpu_supporting_batch_image/

Who has fbgemm.dll error on Windows 11 the solutions posted below topic

https://github.com/Acly/krita-ai-diffusion/issues/965

Quick solution is as below (verified DLL)

Download this dll and copy it to your /windows/system32 folder. https://www.dllme.com/dll/files/libomp140_x86_64/00637fe34a6043031c9ae4c6cf0a891d/download

25 August 2024 Update V10

Cut off at last complete sentence feature added

Multi GPU-batch caption development in progress

24 August 2024 Update V6 Big Update

New library added - huge speed up

4-bit loading added - works perfect - try both and see if works fast

Overwrite previous files and run installer again

More control options added Max New Tokens, Do Sample, Top K, Temperature, Padding, Truncation, Add Special Tokens

Currently they are set as default

Now app will auto remove |. from text outputs

Batch processing errors fixed and logic improved and

24 August 2024 Update

Every single processed image now will be saved into outputs folder with generate caption automatically

Open outputs folder button added

Just overwrite previous app.py file

Input image height set to 512px so you can see better

Now it will show batch processing status better

Also it adds "This is" to beginning of every caption and it will be auto removed now

Windows Requirements

Python 3.10, FFmpeg, Cuda 11.8, C++ tools and Git

If it doesn't work make sure to below tutorial and install everything exactly as shown in this below tutorial

https://youtu.be/-NjNy7afOQ0

How to Install and Use on Windows

Extract the attached Joy_Caption_v12.zip into a folder like c:/joyv7

Then double click and install with Windows_Install.bat file

It will generate an isolated virtual environment venv folder and install requirements

Then double click and start the Gradio App with Windows_Start_App.bat file

When first time running it will download models into your Hugging Face cache folder

Hugging Face cache folder setup explained below

https://www.patreon.com/posts/108419878

It will by default save every caption with options as you set

However for single files sadly Gradio can't read the folder accurately

So use batch folder processing

How to Install and Use on Cloud

Follow Massed Compute and RunPod instructions

Usage is same as on Windows

Regarding Default Expert Settings

App Screenshots and Examples

 

 

 

Files

Comments

Diggy Dre

I keep getting this error: raise ValueError("Unable to infer channel dimension format") ValueError: Unable to infer channel dimension format

Thomas

Batch processing still doesn't work for me, even after reinstalling v2. When I click, nothing happens. And if I try to caption a single image I get "Tensor on device cuda:0 is not on the expected device meta!" although it worked fine on v1. I will try to delete completely the folder and models and reinstall.

eyal giron

the Joy_Caption_v1.zip gives Attachment with id 20240035 was not found when trying to download

Thomas

after completely deleting v1 and installing v2. No error during installing, but after starting if I load an image I get :

Thomas

ERROR: Exception in ASGI application Traceback (most recent call last): File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 406, in run_asgi result = await app( # type: ignore[func-returns-value] File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 70, in __call__ return await self.app(scope, receive, send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\fastapi\applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\middleware\errors.py", line 186, in __call__ raise exc File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\middleware\errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\gradio\route_utils.py", line 760, in __call__ await self.app(scope, receive, send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\middleware\exceptions.py", line 65, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app raise exc File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\routing.py", line 754, in __call__ await self.middleware_stack(scope, receive, send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\routing.py", line 774, in app await route.handle(scope, receive, send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\routing.py", line 295, in handle await self.app(scope, receive, send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\_exception_handler.py", line 64, in wrapped_app raise exc File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\routing.py", line 75, in app await response(scope, receive, send) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\responses.py", line 348, in __call__ await send( File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\_exception_handler.py", line 50, in sender await send(message) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\_exception_handler.py", line 50, in sender await send(message) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\starlette\middleware\errors.py", line 161, in _send await send(message) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 510, in send output = self.conn.send(event=h11.EndOfMessage()) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\h11\_connection.py", line 512, in send data_list = self.send_with_data_passthrough(event) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\h11\_connection.py", line 545, in send_with_data_passthrough writer(event, data_list.append) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\h11\_writers.py", line 67, in __call__ self.send_eom(event.headers, write) File "C:\Users\thomas\Downloads\Joy_Caption_v2\venv\lib\site-packages\h11\_writers.py", line 96, in send_eom raise LocalProtocolError("Too little data for declared Content-Length") h11._util.LocalProtocolError: Too little data for declared Content-Length

Thomas

same. I'll wait for V3 or test on RunPod then

Thomas

It does work on runpod!

Sugar Coat VFX Design

Thanks for quick update~ Can you explain the usage of "Max New Tokens, Do Sample, Top K, Temperature, Padding, Truncation, Add Special Tokens"?

Koudkunstje

Would be nice if the description fits the max token size and doesn't cut of half way the description.

楠 陈

Traceback (most recent call last): File "E:\Joy_Caption_v9\app.py", line 3, in from torch import nn File "E:\Joy_Caption_v9\venv\lib\site-packages\torch\__init__. py", line 148, in raise err OSError: [WinError 126] The specified module cannot be found. Error loading "E:\Joy_Caption_v9\venv\lib\site-packages\torch\lib\fbgemm.dll" or one of its dependencies.

puk

Thanks for the app i will try it out, but i have a question: for a sdxllora training we have always done short captions with WD14 captioning. Is it different with Flux? Does WD14 captioning make no sense here, but rather a full description of the image for the training? I made a test Lora with Civit AI and used WD14 captioning there. I found the result good, will it be better with a detailed description?

puk

THX it works, but how many tokens i can use for training? More than 75? The default setting in your app is 300. And make it sens to insert the trigger word in the textfile

puk

Ok, Is that necessary to train a person? Ohr more less and specific?

Dmitry

Hello, could you add the ability to additionally get tags to photos based on the received promt or image with saving to a csv file according to a given template?

Fossil Bluff

Really? You're telling us to ask chatgpt for answers? How about a little tooltip next to the section heading?

Nikolaj Sørensen

Would it be possible to be able to load other LLMs? Like Mistral Nemo or other finetuned models?

Dmitry

Hello, I will try to explain in more detail: When you upload the generated images to photostocks, you need to assign tags to them so that other users can easily find your work on the site. If you have a lot of generated images, typing tags manually may not be very convenient. The artificial intelligence built into photo stocks doesn't always suggest the right tags, so you have to edit them. I don't really want to use paid services One option is to write tags for each image in the CSV file and upload this file to the photo sheet. This saves you from having to add tags on different photo stocks every time. However, this approach makes sense if the tags in the CSV file were automatically generated. Each photo stock has its own templates for filling in CSV files, so it is advisable to be able to customize the file filling templates. Is it possible to set up an AI that writes a description for images so that it automatically generated tags and writes them to a CSV file using the specified template?

agustin perez

hi! I'm trying to change the host with an argument in Windows_Start_App.bat. Do you know what's the correct argument?

agustin perez

I have added the arguments --host and --port. Talk me if you want it and I'll share to you

Everett Sohn

Is it possible to amend the LLM caption prompt?

juggz143

You should create a readme with an explanation of what all the options do. Specifically "padding, truncation, and add special tokens", the rest are pretty self explanatory. Hover over tooltips would be even better.

Ron Harris-White II

Any idea on how to add a custom token into the caption? (ie - woman vs wife woman)

Mark Fenlon

Prompt padding basically adds empty tokens to the end of your prompt so if there is a discrepancy with the tokenizer you have a buffer and can absorb those additional tokens without cutting off your prompt. Truncation (I believe, I'm fairly sure this is equivalent to 'trim') is where if you run out of response tokens in the middle of a generation, the incomplete sentence is deleted from the response before it is delivered to you. No unfinished thoughts, basically. Special tokens are mostly for the llm itself and response debugging, in that they are instructive short hand for its regular operations like, 'begin your response' or 'stop responding'. There is zero need for any of these settings to be fiddled with in this captioning use-case.

Steve

I know you said you use this for training and use verbose tagging now. I might have missed it but how does koyha deal with periods Vs commas?

Andre Bopp

Thank you very much for this tool. Could you please add a text input that allows you to customize the default captioning-prompt (Variable: VLM_PROMPT)?

Koudkunstje

You know what would be a nice feature? If you could replace certain words by 'your_secret_token'. Lets say you train on a hat, then you could replace 'hat', 'cap', 'head wear', into 'your_secret_token' in the generated caption.or place 'your_secret_token' in front or after the captioning text (if you train for styles or something like that). Yes i can do that with notepad++, but it would be awesome if it could be done in a one click solution.

Torcello

I attempted to make captions using JoyCaption v13 - but after the shards file is downloaded - my system just hangs?!

cool1

In the video I didn't see a "caption prefix". BLIP captioning in Kohya has a caption prefix and suffix option, eg. if all the images in the folder are the same person you could put "a photo of [person name]" in the prefix. That will allow it to be more specific. If there's not that option could one be added please?

cool1

Okay thanks. I'll download that too. I assume I'd then create the batch of captions using the JoyCaption UI, then use that other UI to add the prefix to each one. That should do it. Though if the prefix option was also in the Joycaption UI it would save using two stages/apps for that (it could be done all in one go, a bit like BLIP caption option in Kohya except better, since with BLIP captions the captions aren't as good I think, and I only get about a sentence from that but Joycaption ones (from a demo I used) seem much more in-depth (even though they can be inaccurate). It's still downloading the shards for Joycaption UI so I haven't tested it locally properly yet.

cool1

Thanks a lot. Also when it generates the caption it also saves the image (and for single captions it saves it to the current drive with no set-up for where to put them if you wanted it changed). You might not always need it to save the image. Maybe a checkbox could be added for whether to save the image(s), as you might already have the images at a size you want to use for training.

cool1

Just a suggestion if it wasn't too much trouble to do. In the JoyCaption UI screen, where it says "Input Folder for Batch Processing" and "Output Folder for Batch Processing (Optional)". If they're long paths it would make it easier if you could click to select the folder paths within the UI (without entering them as text), and have an option to make the output folder the same as the input folder that you selected. Sorry if I've made too many suggestions though.

cool1

Thanks a lot. I've seen buttons for folder selection on the Kohya SS screen but maybe they did extra things and it wasn't a standard gradio thing.

cool1

Thanks for the update. I''ve tested the precaption and that works okay and test the "don't save processed image" and that seems working okay too, thanks.

cool1

Thanks a lot. They are working okay. I've tried testing the prefix and suffix and the new "don't save image" checkbox (and prefix with the batch option).

Ammar Salem

I feel so lazy as I am the only one looking for the one click update ;-)

JustLookingForNothin

Hi, PyTorch 2.4.1 Release, bug fix release is out almost 2 weeks and should prevent the need to download libomp140.x86_64.dll from some shady website. Maybe you should update your installer scripts? The latest Joy Caption V15 installer still points to PyTorch 2.4.0. Thanks!

kama

First thank you very much for your software and your tuto that help me very well. I would like to know if it s possible to switch for an other llm like mistral nemo that would be more appropriate? thanks for your response

kama

yea for that moment but it run with llama 3 and it would be better with mistral nemo

cool1

I've noticed that sometimes in JoyCaption, with the "max new tokens" set to about 300 or more, it can repeat the last sentence or if it's about 400 it can repeat the last 2 sentences multiple times (maybe to reach the "max new tokens" size when it doesn't know what else to say). I assume if the repeats were left in it might give too much importance/over-train those repeated sentences. Could the app detect that if possible please and remove any repeats or would it be too complex/too much work? Also the app's name in the Firefox tab name is "Gradio". Though if you have multiple gradio apps it might not be that clear. Maybe the app caption for the browser tabs could be changed to something about "JoyCaption".

cool1

Thanks a lot. I didn't know if the first one would be too complex. I'll download and try it now. edit: Yes it's working okay thank you.

Myk

i have been trying to install this so many times and always get errors when installing it: ModuleNotFoundError: No module named 'huggingface_hub'

Myk

i got it working but i had to continuously press the install bat. NOW i huave an issue where it keeps trying to do this online. I have a local based system and DO NOT want to be online (limited bandwidth). So why cant i get this thing working offline?

Myk

it was downloading shards?? what? it is supposed to work after downloading the model, but now it wants to do this online, and also dumps the files in my windows TEMP folder, which is unsafe

Myk

this was supposed to be an easy solution to caption my dataset before i started training my flux model, but now i have to figure out WHY there isn't clear settings where i can CHOOSE where everything goes. There should be NO REASON this program DUMPS my training images into windows TEMP folder.

SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts

you must be doing something wrong. first of all it saves images into your outputs folder inside locally installed folder. you can also decide outputs folder as well when doing batch. also it downloads shards into your hugging face default cache folder. and default settings works best you dont need to do anything

Myk

it is installed, but when i run the batch it starts to download shards>???

Myk

when i select an image from my folders, it will place that image in my WINDOWS TEMP folder under gradio

Myk

i wonder if it;s because i selected gradio share (option 3), i selected option 1 this time but it started the SHARD download again???? safetensor 0002 of 0004, LARGE FILES?

Myk

wy is is downloading these large 5gb files?

Myk

i dont understand this>>> Why did it download 15gb of shards for this, also where did it download to?? example, model-00002-of-00004.safetensors and 4 others totaling 15gb, and didnt say where they went, as i cant see them in the directory

Myk

well i think i was freaking out for nothing, apologies....but i was honest about those issues. When loading it, i had to repeatedly hit the .bat to install it, (THEN also repeatedly hit the other load bat for the UI) as when it hit an error, usually saying huggingface wasn't there until i had to drag it from my User/python location over to the venv location inside the Joy-Cap 17 Lib... Then if you drag or place an image (was testing just one image) into the load area, it would COPY what you placed and place it in the windows TEMP folder under gradio/(large and long named other folder)/ which i think is not cool at all. dumping training images into an OPEN area is not ok with me at all. Just for safety sake, nothing crazy just concerned.

Myk

last question, do i git pull anything, or do i have to totally reinstall this everytime there is an update? AGAIN thank you soooo much for your tutorials and instructions! your vids are great!

Myk

legend among us people! Thank you! So glad i subscribed! A lot other patreon and you'tubers don't put out the quality i've seen so far from your tuts and instructions! Well done!

楠 陈

Please add Alpha One release

楠 陈

The new version does not remove: "This image is"

Orpheus_Reup

Hi I get this error ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them. torch==2.4.1 from https://download.pytorch.org/whl/cu124/torch-2.4.1%2Bcu124-cp310-cp310-win_amd64.whl#sha256=db873d7c9f4b959ea818780736ad55c3b120d0337b12ef84a0ad5e9a6fc6b147 (from -r requirements.txt (line 4)): Expected sha256 db873d7c9f4b959ea818780736ad55c3b120d0337b12ef84a0ad5e9a6fc6b147 Got 74ff1129e87e85bf9c6c47e8bbf5a02c22bc00c3589ab5ef3526c1397f79c413

Oui

Can you add Alpha Two ? A README with command to use it without the GUI would be awesome also :) EDIT : To try it locally $ git clone https://huggingface.co/spaces/fancyfeast/joy-caption-alpha-two $ cd joy-caption-alpha-two $ python -m venv venv $ .\venv\Scripts\activate $ pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124 /////You can check for different CUDA here https://pytorch.org/get-started/locally/ $ python app.py (I was needed to $ pip install spaces and $ pip install protobuf But I don't know if it's the same for everybody, and btw you can access different llm on duck.ai to troubleshoot)

cool1

In JoyCaption UI, once you've started the batch mode and it goes through each file in the folder (which can take quite a long time depending on number of files), currently if you've made a mistake you need to wait for it to complete all files in the folder. Would it be possible to add a "stop" or "cancel" button please so you could abort it part way through (eg. if you've forgotten to add the caption prefix or some other setting)?

cool1

Okay thanks I'll do that (ctrl+c in the command prompt window then then double click on the start .bat file). Kohya has a stop button (eg. to stop training part way through) but if it's a difficult thing to add that's okay. I don't know if adding a checkbox for stopping the batch would be easier, so it didn't need to stop immediately but could check that value in the gui just before it starts captioning the next file in the list and if the value is true/checked break out of the batch captioning loop. But if that would be difficult that's okay.

cool1

Thank you

Ying Mo

Using cached https://download.pytorch.org/whl/cu124/torch-2.4.1%2Bcu124-cp310-cp310-win_amd64.whl (2506.2 MB) ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them. torch==2.4.1 from https://download.pytorch.org/whl/cu124/torch-2.4.1%2Bcu124-cp310-cp310-win_amd64.whl#sha256=db873d7c9f4b959ea818780736ad55c3b120d0337b12ef84a0ad5e9a6fc6b147 (from -r requirements.txt (line 4)): Expected sha256 db873d7c9f4b959ea818780736ad55c3b120d0337b12ef84a0ad5e9a6fc6b147 Got 3d880258ee1ea10e8b107466d19766c641695b61fee202c133a4b732c2fe8eca

Brian Skipper

Got this error when when I ran this for the first time. Seems there is a backup but maybe it's not as good? Is this something I can manually do? Loading VLM's custom text model Error loading custom text model: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like meta-llama/Meta-Llama-3.1-8B is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'. Checking contents of text_model directory: - adapter_config.json - adapter_model.safetensors - README.md Falling back to default model

Nicolas Nicolau

Can you add a stop button, for the batch processing

cool1

Thanks for adding the new Stop Batch Processing option.

Ant-2014

Thank you for your powerful and actual works in AI themes. I was/am your subscriber long time and I see how you do very interesting, useful works and comparison testing. ... I am waiting also the joy v2 :) and many others projects in this AI chaos in our world! https://huggingface.co/spaces/fancyfeast/joy-caption-alpha-two Best regards Anton Finland