I Am Researching Multi-GPU Training Speed Difference (Patreon)

Rented this machine on Massed Compute to test Stable Diffusion XL training speed difference with Kohya SS GUI on a 8x A100 (SXM4) machine and see the difference of multi-GPU training SXM vs PCIe linked GPUs

Hopefully will update this post as I get more info.

Update 1

Single GPU works with Batch size 7 but more than 1 GPU fails

Reported Kohya : https://github.com/kohya-ss/sd-scripts/issues/1434#issuecomment-2244049699

Single GPU training speed as below with batch size 7