ELLA - Making prompts better! (Patreon)

"The horizon shimmered with a kaleidoscope of hues, a huge towering, iridescent crystal structure erupts from the ground, their facets refracting the vibrant pink and indigo sky. Amidst this otherworldly landscape, a lone figure with his back to us stands wearing an old, tattered, black cloak and a sense of ethereal grace."

Video guide: https://youtu.be/_Pr7aFkkAvY

ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment

CLIP gets stuff wrong all the time! Ask for a blue bear with a red hat, and chances are you'll get a red bear with a blue hat. Not so much with ELLA! Now you can provide long and detailed prompts and actually get what you asked for. Just don't ask for perfect hands ;)

ComfyUI guide:

Install https://github.com/ExponentialML/ComfyUI_ELLA using ComfyUI manager

Models should be saved under your ComfyUI/models directory

The first model to download is the ELLA one, which you save to ComfyUI/models/ella

https://huggingface.co/QQGYLab/ELLA/blob/main/ella-sd1.5-tsc-t5xl.safetensors

The second download is flan-t5-xl, which you save to ComfyUI/models/t5_model/flan-t5-xl

Original t5 = https://huggingface.co/google/flan-t5-xl/tree/main

Smaller t5 = https://huggingface.co/Kijai/flan-t5-xl-encoder-only-bf16/tree/main

Command to download all files:

git clone https://huggingface.co/Kijai/flan-t5-xl-encoder-only-bf16

(Optionally rename that directory to flan-t5-xl)

Original repo guide:

URL: https://github.com/TencentQQGYLab/ELLA

conda create --name ella python=3.10

conda activate ella

git clone https://github.com/TencentQQGYLab/ELLA.git

cd ELLA

pip install fire gradio==3.50.2 torch torchvision diffusers transformers accelerate sentencepiece

python inference.py test test ./ella-sd1.5-tsc-t5xl.safetensors

NB. Gradio 4.26.0 doesn't work due to deprecated functions

Links:

https://ella-diffusion.github.io/

https://github.com/TencentQQGYLab/ELLA

https://github.com/ExponentialML/ComfyUI_ELLA

https://huggingface.co/QQGYLab/ELLA/blob/main/ella-sd1.5-tsc-t5xl.safetensors

https://huggingface.co/google/flan-t5-xl/tree/main

https://huggingface.co/Kijai/flan-t5-xl-encoder-only-bf16/tree/main