Home
Creators
Search Recent Random
Posts
Search DMs Popular Hash Lookup Tags Random
Importer
Import FAQ
Register Favorites Keys Review DMs Logout
Community
Telegram ThePornDude
Home Creators Posts Import Register Favorites Logout

Simple LLM Enhanced Flux! (Patreon)

Published:
2024-10-12 13:07:48
Edited:
2024-10-18 11:14:21
Imported:
2026-06
Tags:

Downloads

Content

Video Guide: https://youtu.be/qMNJ0dCoT_I

This is one for those who don't like noodles, tweaking things - or even moving around in the ComfyUI interface!

Everything you need for generating is all in once place, with shortcuts keys for the first-time use settings. This leaves you to focus (foooocus?) on prompting, with just the required options for:

Prompting

Image size

Seed

LLM prompting mode

HighRes mode

USD mode + resize factor

Example interface screenshot:

Press 1 = Model Settings

Press 2 = LLM Settings (optional)

The custom scheduler is a sort of mix between beta & simple which seems to gives the fewest stripes whilst also having reasonable detail. I've tested up to 2048x2048, so you can get fairly large outputs. Note that Ultimate Upscale takes a while!

Update Notes!

13/Oct/24 - The updated version (Flux-Simple-LLM-Ultimate_v2.json) has better switch logic, so now turning multiple things on or off at the same time should work properly :)

18/Oct/24 - Added Flux-Simple-LLM_v1.json, which is much the same as v0 but has a power lora loader and the LLM party node has been updated.

NB. The LLM party node was updated recently such that you'll need to:

Copy the existing system prompt text in the LLM Party node

right-click and select "Fix Node (recreate)" on any older workflows with this node.

Paste the system prompt text back in

 The node has already been re-created in v1.

The same fix has been pre-applied to Flux-Simple-LLM-Ultimate_v2a.json

LLM Prompting Mode

The prompting mode is configured slightly differently here to enable "conversations". You can chat with Llama3.2 making prompts such as these possible:

Add more flowers

Give everyone a silly hat

What is to the left of them?

What is the main character thinking about?

Ollama

For the LLM Enhanced workflows you'll need to have ollama installed & running:

https://github.com/ollama/ollama

The workflow is configured to use llama3.2 by default, which you can download by running:

ollama pull llama3.2

Workflow Notes

Three different workflow versions are attached to this post:

Flux-Simple_v0.json = Simple interface, no LLM

Flux-Simple-LLM_v0.json = Simple interface + LLM

Flux-Simple-LLM-Ultimate_v0.json = Simple interface + LLM + Ultimate Upscale

If "Install Missing Custom Nodes" fails, they can be installed by using the search option instead:

Impact Pack

Extra Models for ComfyUI

ComfyUI-GGUF

ComfyUI_SUNoise

UltimateSDUpscale

Comfyroll Studio

rgthree's ComfyUI Nodes

ComfyUI Extra Samplers

Use Everywhere (UE Nodes)

KJNodes for ComfyUI

ComfyUI-ppm

pythongosssss/ComfyUI-Custom-Scripts

comfyui_LLM_party

Models

If you are missing any models, download them from here:

Flux = https://huggingface.co/black-forest-labs/FLUX.1-dev

Flux GGUF = https://huggingface.co/city96/FLUX.1-dev-gguf/tree/main

CLIP = https://huggingface.co/zer0int/CLIP-GmP-ViT-L-14

Hyper-SD = https://huggingface.co/ByteDance/Hyper-SD

Shakker Labs = https://huggingface.co/Shakker-Labs

Upscalers = https://openmodeldb.info/

Low VRAM

A GGUF loader (bypassed by default) is also available for those with low VRAM. Simply delete the standard loader then un-bypass and connect the GGUF one. The default configuration works best with 24GB+

Files to download for the regular version

From https://comfyanonymous.github.io/ComfyUI_examples/flux/ -

"If you don’t have t5xxl_fp16.safetensors or clip_l.safetensors already in your ComfyUI/models/clip/ directory you can find them on: this link. You can use t5xxl_fp8_e4m3fn.safetensors instead for lower memory usage but the fp16 one is recommended if you have more than 32GB ram.

The VAE can be found here and should go in your ComfyUI/models/vae/ folder."

Files

Comments

Joe Blow

I've updated my ComfyUI but don't see an option for the new interface. Love the content BTW.

Raja Nandepu

nice do we have img to img - mode with upscale ?

NerdyRodent
>>152181968

The hires mode does indeed essentially use img2img for the upscale. Should modify a few details (hopefully for the better) along with the increased resolution!

David Henrry Ticona

Hi Nerdy, trying to download the workflows, it says "Expired URL" for these links: Workflow Notes Three different workflow versions are attached to this post: Flux-Simple_v0.json = Simple interface, no LLM Flux-Simple-LLM_v0.json = Simple interface + LLM Flux-Simple-LLM-Ultimate_v0.json = = Simple interface + LLM + Ultimate Upscale can you please share them again?

Nibmeister

Flux-Simple-LLM_v0 works really nicely on my 3070TI.

SouthbayJay

Thank you for the workflows and the video! For some reason I'm not getting the new UI.. I have the bar on top and side but not the floating queue button or the models folder.. Anything special I need to do?

SouthbayJay
>>152447548

I have the new version but I'm missing a lot of the features.. Like the models folder on the left side bar, the default template menu and the floating queue button... I've updated a few times but those don't seem to appear nor do I see anything in the setting menu.

droghtak

A 3090 is having hard time with this one, I need to load the gguf version which is a shame. I highly suggest to try the nf4 models, they work similar or even better than FP8

NerdyRodent
>>152533309

I'm on a 3090 here and it peaks up to 18GB VRAM, but with an initial image of 2048x2048 and HiRes enabled, it can go up to 19GB. Personally I avoid the GGUF models as they're so much slower.

droghtak
>>152533309

MMM I'm using your ultimate V2 workflow on Ubuntu. Perhaps I'm doing something wrong but I'd say I'm using the same on everything. Are you using a headless system? Perhaps my X are eating too much VRAM

NerdyRodent
>>152533309

Oddly enough, the Q8 GGUF actually uses more VRAM for me (as well as being slower XD). I've got two monitors attached, with my pre-comfy VRAM usage sitting at just under 400MB (225 standard, 300 if I open Chrome). Never had an OOM yet (even with larger LLMs)!

droghtak
>>152533309

Damn it I have to recheck everything. I guess you are using regular comfiuy from official repo right? Perhaps I gave some troublesome node 🤔

Loakamts
>>152533309

It’s also possible to use openrouter as an llm supplier. While it does cost a bit, its pay as you go (and not expensive depending on the llm your using). I understand privacy might be a concern though. If you’re interested in the setup, let me know (it’s pretty quick and easy). 👍

B. Elliott Baker

trying to use this but after i install Comfyroll Studio nodes my comfyui won't load the workflow... i just get a spinning wheel forever.

NerdyRodent
>>152772706

You can either start over with a fresh environment, or debug your existing one. If there are any errors shown while comfy is starting up, that would be the place to start. Often a standard install can have fewer issues in the long run than the portable download, if you're using that. Anything outside the standard Linux + Nvidia setup can come with it's one, special issues too!

NerdyRodent
>>152447548

If you're using the legacy interface then you'll need to turn the new menus on via the settings cog.

droghtak
>>152533309

okey just for completion, I was using a FP8 model with vae and clips backed in... im so smart :D. Besides this, sometimes chrome hoards gazillions of VRAM for no reason. But everything is ok now :-)

simon gentry

hi everyone! i'm new here... when i load in any of the node setups i get Missing Node Types and a LONG list of missing nodes, which don't seem to be found by the Manager custom Node search engine... any thoughts? there are about ten which show no results. many thanks!