Home
Creators
Search Recent Random
Posts
Search DMs Popular Hash Lookup Tags Random
Importer
Import FAQ
Register Favorites Keys Review DMs Logout
Community
Telegram ThePornDude
Home Creators Posts Import Register Favorites Logout

Most Powerful Vision Model CogVLM 2 Gradio APP Published, Caption Images and More, 1-Click to Install on Windows, RunPod and Massed Compute (Patreon)

Published:
2025-01-17 13:53:57
Imported:
2026-06

Content

Info

Latest installers : https://www.patreon.com/posts/120193330

Official repository of the APP : https://github.com/THUDM/CogVLM2

This app above is purely developed by me

CogVLM 2 is GPT4 level for vision tasks and it is open source and free

Check above screenshots to see how to use it

Currently the APP works amazing with 4-bit quantization very fast

I am searching to lower VRAM usage even further with like adding CPU-Offloading and other stuff if possible

Previously we were lacking Triton but it now works perfect

If you get Triton error make sure to delete your Triton cache after installing the app like below

C:\Users\Furkan\.triton

Instead of Furkan your username 

The app has so many features so checkout the screenshots

1-Click installers will install into Python 3.10 VENV completely secure and isolated

 

 

Files

Comments

BecauseReasons

So, apart from captions what's the point of this?

TD Film Studio

I have been using Cog VLM 2 since your earlier releases, without proper triton compatibility on windows. Even though it was slow, I loved the results. This upgrade is very good news and I thank you for the upgrade

jamzsam

It says in the Repo that they released the CogVLM2-Video model. Does this include it. I'm busy training loras for Hunyuan video and a good video captioner would be great.

jamzsam

Thanks Dr Furkan. Im getting amazing loras results for Hunyuan video. You can train image and videos. I had to install the trainer through WSL Linnux sub system for windows which was a headache to figure out. On my RTX 3090 I'm able to get 1280x720, 5 seconds at 24fps in comfyui using the kijai nodes ;)

cool1

Is this more accurate than the JoyCaption versions? One thing I noticed in the examples is, for photos it doesn't seem to say it's a photo. eg. it says "the image captures a snowy scene" or "the image features a man" but not "a photo/photograph of...". So maybe that could make the captions less specific. Though it does say for things like "digital artwork" for some other images.

Ahmet Inceelli

Hello, Is it better than Joy Caption? 😊

jamzsam

Nice! I've noticed One trainer has added Hunyuan training but I'm not sure you can train video clips. Only images. :)

cool1

I tried this yesterday and it worked but with 1 24 GB VRAM GPU it got really slow trying to use the non-quantized version. Maybe it takes about 40 GB of VRAM/RAM (based on the total size of the model files/shards it downloaded which were about 8x ~5 GB) which is over my card VRAM. Or maybe it was also because I also had other apps loaded at the same time or because I increased the amount of tokens to create. Maybe the menu where you select which version to load (unquantized or the different quantized versions) could say how much VRAM each uses. When I turned the router off after I'd pressed the caption button it said something like "Connection Error" in the UI. Why though if it's supposed to be running locally and the models used were already downloaded? Is it because it (or some part of gradio) is logging something and if so why? Or is it because in the code in the get_model section it says "trust_remote_code=True". Does it need that because the model is non-standard and does that make it need the internet even after the model is loaded? Is that a security risk when we can't see what remote code is being run? Also starting captioning it checks and downloads new versions of python files if they exist. Couldn't it just keep the versions that were already there unless you ask it to update them (eg. there's a warning that shows that says "Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.". I don't know how to pin a revision but it would probably be better if there was an option for whether to update. Also the browser tab for this shows "Gradio" as the title but that's the same title as the browser tab for Joy Caption Alpha2. If the browser tab was titled something like "CogVLM2 UI" it would be easier to distinguish each. I just tried the 8 bit quantized version and that also seems very slow (about 13 mins for 1 image) and using a lot of VRAM too. Maybe it's a much bigger model than Joycaption (maybe it's checking at a higher res too so could be more accurate eg. for text). With the 4 bit version it works reasonably fast (about 62-106 secs for creating with max of 2048 tokens, maybe depending what other things are running). So if it's more accurate than JoyCaption that could be used. Though at 2048 tokens max it only produces (in the tests I'd done at the time between 151 and 183 words, which is less than the 260 word count that JoyCaption can output (perplexity says 2048 tokens should be about 1500 words)). Maybe there's some cut off it's using (other than the 2048 token limit) that limits the number of words that might be able to be used. Though for small text in the image COGVLM2 seems much better than JoyCaption, since it described what was being said in 4 lines of text in an image (though it didn't quote it exactly first time, but if you ask it to quote it exactly in the prompt it sometimes does but didn't when I asked it for other more detailed info about the image too. Though asking it that also makes it hallucinate sentences of text that don't exist in the image sometimes.). Another thing that would be good if possible is if, when you want to put a different image into the UI when there's already one there, it would be good if you could just drag the new image over the old one without pressing x to close the original one. If you drag it without closing the original one it just shows the image in the browser not as part normal UI.

SECourses: FLUX, Tutorials, Guides, Resources, Training, Scripts

yes this is a very big model. that is why it is slow. biggest image captioning model we have atm . about your last thing i dont know i am using gradio default image editor maybe they need to support ye i will check app title and i dont know your internet related question. and you have to use 4 bit on 24 gb gpu

楠 陈

Encountered exception while importing triton: DLL load failed while importing libtriton: The dynamic link library (DLL) initialization routine failed. Error processing image: DLL load failed while importing libtriton: The dynamic link library (DLL) initialization routine failed.