Machine Learning Image Upscaling in… Houdini! (Patreon)

Starting with version 20.0, SideFX has started adding many machine learning tools to Houdini, including support for ONNX models. Let’s have a look and create an implementation of the Gaffer image upscaling workflow we covered here: https://www.patreon.com/posts/121405052

Upscaling a Single Image Tile

Just as we did in Gaffer, we’ll use the ONNX Super Resolution model: https://github.com/onnx/models/tree/main/validated/vision/super_resolution/sub_pixel_cnn_2016 In order to apply the model to any image we want we’ll use Houdini’s new Copernicus toolset.

Copernicus is set of procedural image creation and compositing tools, intended as a replacement for the old COP2 workflow. COP stands for Compositing OPerators, in keeping with Houdini’s context naming conventions. COP2 was (is) ancient and wasn’t particularly flexible or well-liked, but Copernicus, while quite new, is generating a lot of excitement in Houdini world. It allows for complex texture creation similar to Adobe Substance Designer. Plus, it has support for ONNX models, which is what we’ll be focusing on here.

To get started, we’ll import an image we’d like to upscale. I’ll use the same butterfly render as in my Gaffer example. Starting small, let’s take a single tile from the image and upscale it. ONNX Super Resolution requires an input image of 224x224 pixels, so larger images will need to be broken into multiple tiles first.

Let’s upscale the butterfly’s eye to showcase how well the model retains fine detail.

Navigate to the /img context and dive into copnet1.

Add a File node and open an image of your choosing. Select the Composite View tab and enable the Display flag on the file node. This will show our work in a 2D compositing view instead of the normal Houdini 3D viewport.

Attach a Crop node and position the crop as desired, with the Upper Right parameter +/- 224 pixels from Lower Left. In this case I used an expression to automatically keep the crop at the correct size no matter where I positioned the lower left. For this 1920x1080 image, the bottom left coordinates are 0, 0 and the upper right are 1920, 1080. I placed the lower left of the crop at 500, 600 to line up with the butterfly eye and then the expression sets the upper right to 500 + 224 and 600 + 224.

Next, we’ll need to convert the image from RGB to YCbCr, per the ONNX model requirements. Attach a Wrangle and add the following code:

f@Y = (.299  @C.r) + (.587  @C.g) + (.114 * @C.b);f@Cb = 128 - (.168736  @C.r) - (.331364  @C.g) + (.5 * @C.b);f@Cr = 128 + (.5  @C.r) - (.418688  @C.g) - (.081312 * @C.b);

@C.r = @Y;@C.g = @Cb;@C.b = @Cr;

The @C attribute is Houdini’s hard-coded color parameter. As expected, it has three components, @C.r, @C.b and @C.g. We declared and assigned values to three new float variables, @Y, @Cb and @Cr by prepending f for float. This RGB to YCbCr formula is the same one I used in my Gaffer article.

Our image now consists of a luminance, Y, channel and two color channels.

Next up, add an ONNX Inference node and load our .onnx file. Click the Setup Shapes from Model button to get everything working.

The ONNX node now has upscaled our image to 672x672. We’ll also implement a normal upscale increasing the original image to 672x672 using a Resample node.

Attach a Channel Extract (or Channel Split) to the ONNX node to isolate the first channel, Y, then combine it with the Cb and Cr channels from the traditional upscale using a Channel Join. We’ve now got a new YCbCr image with the Y channel treated with the ML upscale and the other two channels scaled using traditional means.

Finally, we’ll convert this image back to RGB using another Wrangle. Here’s the required code:

f@finalR = @C.r + 1.402 * (@C.b - 128);f@finalG = @C.r  - (.34414  (@C.g - 128)) - (.71414  (@C.b - 128));f@finalB = @C.r  + (1.772 * (@C.g - 128));

@C.r = @finalR;@C.g = @finalG;@C.b = @finalB;

That’s all there is to it! Our 224x224 butterfly eye tile is now 672x672. We can display this upscaled image and a standard upscale side by side using a Contact Sheet. It’s easy to see how much sharper the ML version is (on the right). It does have the expected edge artifacts, which we’d have to address with a crop node before finalizing our work.

If desired, attach a ROP Image Output to save the resulting image to disk in any format you like.

Upscaling Multiple Tiles

You might remember from my Gaffer article that we used a lot of cool Gaffer features to automate processing many tiles and assembling them into a final, large image. Gaffer’s version of the Contact Sheet is perfect for that.

The bad news is that since Copernicus is so new it’s quite cumbersome to collect, process and assemble a large number of tiles. A Copernicus Contact Sheet can accept any number of inputs, but we’d have to literally paste in 45 File nodes, process them through the ONNX model and then make 45 individual connections to the Contact Sheet. We’ll, it’s possible, but anybody who values their time and sanity is not going to want to bother with this! Even with just 4 connections you can see how awkward this would get:

Probably a better approach would be to treat the initial tiles as animation frames and render out each processed tile as a new frame, then assemble the tiles into a final image elsewhere.

Interestingly, the old COP2 workflow has a Mosaic node that would be perfect for our needs. It works very much like Gaffer’s Contact Sheet and can accept a multi-frame input to populate a “contact sheet.” Hopefully this will be coming to Copernicus sooner rather than later.

You could also consider using Houdini’s automation tools, TOPs (yep, Task OPerators), to automate tile creation, upscaling, rendering and assembly. This is probably the most Houdini-like solution and it would certainly work. However, it’s still overly complicated, especially compared with Gaffer. But, if you wanted to remain entirely in Houdini it’s the way to go.

Conclusion

In this article we dipped a toe into Houdini’s powerful new image manipulation toolset, Copernicus, and explored how to easily implement an ONNX machine learning model to upscale an image. This is just scratching the surface of Houdini’s ML capabilities. I plan on covering this topic much more in the future. Stay tuned!