Home Creators Posts Import Register Favorites Logout

Content

โœจ Hey friends! I'm excited to share this install/start-up guide for StreamDiffusionTD. ( TOX available now in separate post ) This operator is built in TouchDesigner 2023 and won't be compatible with earlier versions! Also, projects saved in 2023 cannot be opened in 2022. See the release post for more - Touchdesigner 2023 Official Update ๐Ÿ›  Installation Tips!! (working on a readme + smoother op install process)Python v 3.10.9 ๐Ÿ’Žis tested and working! Other versions work, but may run into issues with the installation process.+ NDI-SDK, Python (3.10.9) and GIT are required for everything to install smoothly. Installing NDI-SDK (needed for ndi-python) - https://ndi.video/download-ndi-sdk/Adding GIT to PATH on windows - download / install & check add to PATH when asked if already installed and not in PATH - https://www.delftstack.com/howto/git/add-git-to-path-on-windows/๐Ÿ‘‹ Let me know if you run into issues! This is v1 ! Reach out ( Discord is best ) if you have any questions or want to share what you've created with StreamDiffusionTD. Always great to connect with everyone and see how you all use these tools. StreamDiffusion github - https://github.com/cumulo-autumn/StreamDiffusionHappy creating. More coming soon.- Lyell

Files

Real-time diffusion in TouchDesigner - StreamdiffusionTD Setup + Install + Settings

Important ! This uses TouchDesigner 2023 and will not work in TouchDesigner 2022. Important ! You may need to also download + NDI-SDK & install Visual Studio A quick starting guide on StreamDiffusionTD. available here -https://www.patreon.com/dotsimulate StreamDiffusion github - https://github.com/cumulo-autumn/StreamDiffusion Installation Tips: Python v 3.10.9 is highly recommended. Installing NDI-SDK (needed for ndi-python) - https://ndi.video/download-ndi-sdk/ Installing Visual Studio / Desktop development with C++ - (needed for ndi-python) Download from https://visualstudio.microsoft.com/downloads/, make sure to install "Desktop development with C++" Adding GIT to PATH on windows - download / install & check add to PATH when asked if already installed and not in PATH - https://www.delftstack.com/howto/git/add-git-to-path-on-windows/ 00:00 Introduction to StreamDiffusionTD 00:48 System Requirements for StreamDiffusionTD 01:12 Setting up operator in network 01:37 Downloading StreamDiffusion repo 02:10 Installing venv (first install only) 03:04 Setting Model + parameters before Stream 03:30 Step Schedule part 1 03:46 Similar Image Filter 04:00 Starting the Stream / Cmd window 04:41 Disable / Enable to see Stream bug 05:10 Seed 05:47 Prompts 06:16 Setting up Prompt Box 07:10 Negative Prompts do NOT work 07:22 Step Schedule part 2 08:16 Guidance Scale / Delta 08:47 Experiment ! 09:16 Lora loader page 09:33 Callbacks 10:03 Stopping the Stream 10:09 Setting OSC Ports / Stream name 10:18 Visible Window (ON is default) 10:32 Say hey on discord ! 11:00 Thanks !!

Comments

virusvjvisuals

Thank you for this marvellous tool you gave us dude !!!

Yvann

Thanks a lots !!

Max Pechony

hey i seem to have a problem when i push the ndi install on the SDTD1 comp it open the CMD window but says : Encountered error while trying to install package. โ•ฐโ”€> ndi-python any help?

DotSimulate

hey ! just edited the post and added the links for additional needed install stuff. Also seems like python version 3.10.9 is working best.

DotSimulate

Hoping to get installation stuff sorted from feedback soon and should be able to update the operator in order to fix things.

Max Pechony

Thanks :) i think the python was the problem i had a newer version i uninstalled it and installed the 3.10.9 and everything works now :)

Michael Carychao

Got it working with python 3.10.11 -- this is siiiick! Thx

Kawatin

It's very ingenious and greatly stimulates my creativity. Thank you for sharing this! Do you support TensorRT?

DotSimulate

Yep! Tensorrt will definitely work. You can set the acceleration mode under the model parameter. Installation is a bit trickier for that but itโ€™s working for several people. Gives a good 5-10 fps boost.

Kawatin

Thank you for your feedback! It seems very difficult and trickier for me to install, but I will try it. ARIGATO

Esref Berk Turkoglu

Another awesome integration, thanks to you. What a time to be alive!

DotSimulate

Not currently. But is something I am thinking about + looking into.

Jonah Hache

would love that for m1. I'm having a lot of fun with my desktop in the mean time!

Jordan Nering

You freaking did it man!!!! Eeeeeek!!! Iโ€™m so god damn excited to try it!!!! Absolutely blown away!!!!!

tajinder singh

is that work with non commercial version ?

DotSimulate

I have not specifically tested but it should.

Faraz Kh

Hi Thank you for the cool project. I followed every step (and have python 3.10.9 installed) but when I click on Start Stream Palse, I get this error in CMD: File "C:\StreamDiffusion\StreamDiffusion\venv\lib\site-packages\requests\adapters.py", line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)), '(Request ID: 9e4fb7bc-2849-4433-a8ad-ff96428da757)') Model load has failed. Doesn't exist. I have also tried to manually download the repo for stabilityai/sd-turbo on my machine and set the Model ID / Path to the local directory or the safetensor. Neither work. I have also tried your v 0.1.1

DotSimulate

A few other people have run into that. A solve I believe has been downloading the model locally, but seems like you tried that as well. Have you tried setting the model parameter to a local 1.5 model.safetensors file?

Faraz Kh

I am not sure if I am setting the local model correctly, can you tell me if this is correct? 1. Clone repo from https://huggingface.co/stabilityai/sd-turbo 2. In the StreamDiffusionTD, set Model ID/Path to sd_turbo.safetensors in the cloned repo When I do this and click on the pulse for Start Stream, I get this error: File "C:\StableDiff\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 833, in _apply param_applied = fn(param) File "C:\StableDiff\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 1158, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) NotImplementedError: Cannot copy out of meta tensor; no data! Should I do it differently?

DotSimulate

Make sure you are setting the Model ID/Path parameter to the full file path to the .safetensors file for the model you downloaded and I think it should work. j

Faraz Kh

It is not working. I am setting Model ID/Path to: C:/TouchDesigner/sd-turbo/sd_turbo.safetensors but when I hit start stream, I get: Traceback (most recent call last): File "C:\Users\faraz\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 314, in _bootstrap self.run() File "C:\Users\faraz\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, **self._kwargs) File "C:\StableDiff\StreamDiffusion\ndi-stream\main_ndi.py", line 182, in image_generation_process stream = StreamDiffusionWrapper( File "C:\StableDiff\StreamDiffusion\ndi-stream\..\utils\wrapper.py", line 153, in __init__ self.stream: StreamDiffusion = self._load_model( File "C:\StableDiff\StreamDiffusion\ndi-stream\..\utils\wrapper.py", line 429, in _load_model pipe: StableDiffusionPipeline = StableDiffusionPipeline.from_single_file( File "C:\StableDiff\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 864, in to module.to(device, dtype) File "C:\StableDiff\StreamDiffusion\venv\lib\site-packages\transformers\modeling_utils.py", line 2271, in to return super().to(*args, **kwargs) File "C:\StableDiff\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 1160, in to return self._apply(convert) File "C:\StableDiff\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) File "C:\StableDiff\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) File "C:\StableDiff\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) File "C:\StableDiff\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 833, in _apply param_applied = fn(param) File "C:\StableDiff\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 1158, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) NotImplementedError: Cannot copy out of meta tensor; no data!

DotSimulate

The path looks correct. Im not sure exactly what issue would be causing it. Have you tried a path to a local 1.5 model instead?

Faraz Kh

I realized I had a problem with huggingface. Even when I try to clone a model, it tells me the connection has been reset. I connected via a VPN and things are working now. It is a bit weird because even after the initial model download, I still need to remain connected to the VPN, otherwise, I get a rejected request. I hope you publish more tutorials on how to use different aspects of this. Very cool project. Thank you for your support too!

Faraz Kh

It would be great to have some cloud support as well. Maybe replicate? I would be happy to help you out write the python code.

json parser

for those wondering: I had python 3.12.something and things didn't work, uninstalled and re installed the 3.10.9 and everything works great.

json parser

anyone have some tips on how to get better FPS? or it depends mainly on the hardware? (i'm using a NVIDIA GTX 3000 ADA GENERATION, w/ 8GB vRAM)

DotSimulate

Yes on GPU as well as CPU, since the VAE model is run with CPU. tensorrt seems to be the largest improvement in speed, but is a pain to set up. Hoping to get enough feedback on setup from people with that though in order to put together a more useful installation cheat sheet or something for it.

json parser

Apart from the python version issue it seems to be pretty easy, what's the problem with tensorrt?

DotSimulate

Theres just more a few more things to download (cudnn) and it seems an even more sensitive setup. Still gotta figure out what the right recipe is.

Arthur Deleu

This is incredible! I only hit 4 FPS however at max, on an RTX3070-Ti. Incredible nonetheless!

Ilsu Deniz Nurcan

Hello there, Thank you for sharing this file! When I start streaming it gets stuck at "Loading pipeline components...: 60%" What could be the reason?

DotSimulate

Not sure why it would get stuck there. I would try shutting down by closing the cmd window, and then relaunching.

Ilsu Deniz Nurcan

yes i did it many times also restarted my pc many times, i assumed it may be because of lack of storage so i freed up enough space also, any other suggestion?

Alexander Korneev

Hey! What CUDA version are you running at? 11.8?

DotSimulate

Yes. cuda 11.8 and python 3.10.9 I should have noted or had an option for that. if using 12.1, you could manually install via the github, or edit the installStreamDiffusion.bat file to reflect the correct Cuda for your system,

DotSimulate

Try making sure TD is the only other thing open. If you have under 8 gigs of vram, it will likely be bottlenecked by the lack of space.

Alexander Korneev

Will try to reinstall CUDA and start from scratch

Will MacNeil

Hi, I'm getting an error when starting the stream for the first time. imoprt NDIlib as ndi ModuleNotFoundError: No module named 'NDILib' I also noticed that I get an error when running the ndi-python install from the settings: error: legacy-install-failure Encountered error while trying to install package. -->ndi-python So I'm guessing I've got something wrong when I install this package. I've done all the installing through the steps inside the tox file. So I'm not sure if it's possible to have a conflict with Python or any of the other modules. Any thoughts? Thanks!

Adrien Oneiga

hi iโ€™m getting a weird error saying "streamdiffusion" requires a different python 3.9.13 not in >=3.10.0. i tried to install the 3.10.9 as you mentioned above but it seems like it wants 9?

Omar Fajardo

Hello, I am Mexican and I am having problems installing what is necessary for it to work correctly, I don't know if it is the language but I am a little frustrated. I have already installed everything necessary but when I install SD environment it sends me errors and does not allow me to continue

DotSimulate

hey. if you are able to shot a message via discord with screenshots of the error, I am happy to look at it. Patreon is not ideal for the troubleshooting stuffs.

Luciano Ferrarezi

Im getting this error when i try to run `Install ndi/spout req only. The system cannot find the path specified. Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases. Do you know what can resolve that problem ?

mengu gulmen

Great work man -like all your works!- really appreciate this. I run everything just fine. I installed all the missing -but not required- stuff that couldn't be installed automatically, and I'm getting about 5-6 fps on RTX3060 mobile with 6gb vram. Something that I could not figure out though: I used "stabilityai/sd-turbo" as model in the operator parameters. The scripts downloaded the model, but i can't figure out where? not in streamdiffusion/models, there's nothing called sd-turbo anywhere in the directory. where does this one put the downloaded model? :D

DotSimulate

It is stored in huggingface cache. and links with the model based on the Huggingface id. C:\Users\\.cache\huggingface\hub\models--stabilityai--sd-turbo

mengu gulmen

Had a *great* time trying to change the HF cache location :D Could manage it in the end, writing here for posterity, for windows: in the streamdiffusion-td/StreamDiffusion/venv/Scripts directory, edit "activate.bat" and add: set XDG_CACHE_HOME=d:/tmp/.cache/ set HF_HOME=d:/tmp/.cache/huggingface set TRANSFORMERS_CACHE=d:/tmp/.cache/huggingface/hub set HF_HUB_CACHE==d:/tmp/.cache/huggingface/hub (not sure which one worked actually :D)

mengu gulmen

I'm trying to find a way of using a server-client setup and to connect to SD on my RTX pc, while using TD from my macbook. I started the server (pulsed the stream) from the pc, then shut down TD. on the mac, with NDI tools, i can see the output stream. however: a) StreamDiffusion-TD doesn't see the remote NDI stream b) can't figure out how to send the out stream to the remote NDI server. any ideas?

mengu gulmen

I actually have one more question: Did anybody figure out how to run SDXL with streamdiffusion? I tried to wade through the code but I'm really green in terms of the diffusers library. It seems like replacing some StableDiffusionPipeline calls with StableDiffusionXLPipeline would do the trick but some quick tests showed me that it didn't :D I now see that I have another question, is there some "low vram" mode in streamdiffusion so that I can load larger models (like in comfy or automatic webui)? (I now realize that these are actually streamdiffusion questions, I hope I'm not being irrelevant here)

lolgang

Hi where is the link to discord?

lolgang

also I was wondering how to export single images from TD using the above method also can we use safetensors and checkpoints within? I'm sorry for the waterfall of questions but im really new to this

elias muhammad

I am having problems not wanting to work when I start streaming. The CMD page always closes the window automatically, and the information is Launching StreamdiffusionTD. What's wrong with this? Please help.

DotSimulate

Hey ! there is a DEBUG command toggle in the settings 2 page that will keep the command window open when there is an error.

elias muhammad

Thank you for your help, I have succeeded, but I can only run up to 16fps, using an Intel i9 CPU, with an Nvidia RTX 2080 Super graphics card, and a GTX 1660 Super graphics card. become one. Is that the maximum? Is there a recommendation for a graphics card that can reach at least 30Fps?

ricarchelo

I get this error Warning: No CUDA devices available. Defaulting to CPU. Traceback (most recent call last): File "C:\Users\ricar\Downloads\streamv2v-main\streamv2v-main\StreamDiffusion\streamdiffusionTD\main_sdtd.py", line 84, in import cv2 ModuleNotFoundError: No module named 'cv2' [proceso terminado con el cรณdigo 1 (0x00000001)] Ahora puede cerrar este terminal con Ctrl+D o presionar Entrar para reiniciar.