Quick Look: Machine Learning in Gaffer (Patreon)

Newer releases of Gaffer contain a few new nodes dedicated to machine learning tasks. It's all very preliminary and experimental, but it's also pretty interesting and could eventually lead to some powerful workflows.

In this article we'll install ONNX on Windows and run a simple task in Gaffer, created by Lucien Fostier.

First up, the Open Neural Network Exchange runtime (ONNX). Download the Windows x64 build here: https://github.com/microsoft/onnxruntime/releases/download/v1.20.1/onnxruntime-win-x64-1.20.1.zip

Extract the archive to a location of your choosing.

Next, download and extract the archive at the bottom of this article containing the YOLO sample file. It's a very lightly revised version of the original posted by Lucien on his GitHub. I cleaned up the scene a bit to make it more portable and added some royalty-free sample images from Pexels.

We'll also need a Gaffer build version 1.5.2.0 or newer. Feel free to use my Gaffer Installer to make setup a breeze! https://github.com/BrianHanke/gaffer_windows_installer/releases/download/2024.10.0/gaffer_installer.exe

Next, go to your Gaffer /bin folder and edit gaffer.cmd in a text editor. Add two lines:

set GAFFERML_MODEL_PATHS=path/to/YOLO/modelsset ONNX_ROOT=path/to/onnxruntime-win-x64-1.20.1

Save the changes and then double-click gaffer.cmd to launch Gaffer.

From there, load the YOLO.gfr file. Select the ImageReader node and update the path to the images to point to their location on your system.

Lastly, focus the YOLO node and scroll through the timeline to see each of the four images.

The ONNX model looks at each image and identifies the subject. Then, Lucien's Python code processes the results, outlines the subject and tags it. It works quite well. Feel free to add your own images to test it further.

This has been a very early and very simple look into machine learning in Gaffer, but I'm sure we'll be coming back to this topic in more detail in the future!