Deep Images and You (Patreon)

You might have heard about deep images, deep renders or deep EXR. Deep is a method for storing per-pixel depth information in renders of 3D scenes. Deep has been around for 15-20 years, at least in high end productions. Let’s look at how the tech works and what we can use it for!

Deep is somewhat similar to cryptomatte (which we discussed here) in that it assigns pixels in a 2D image to layers. Cryptomatte is limited to “in front” or “behind,” but deep stores the precise position of each pixel in 3D space.

What is the benefit of this? After all, we already have access to Z (depth) AOVs that promise the ability to sort pixels by position. There’s a big issue with Z though: it doesn’t account for transparency. Motion blur, edge anti-aliasing and transparent objects like glass or smoke aren’t manageable. For example, if you have an object with motion blur, what is the depth position of the blur? It “belongs” to the object in motion, so Z assigns the blur a position of whatever object it’s connected to. The problem is that there are also objects in the background that show through the blur and Z has no way to represent that.

Deep does. A deep render, similar to cryptomatte, accounts for transparency by storing pixels in layers. A totally opaque pixel needs only one layer (although the depth position is still stored), but a pixel with transparency is broken into however many layers it needs for each object visible in the same pixel.

This has benefits for compositing since you can slip images in and out of various positions in 3D space. Obviously, compositors are happy to have the ability to do this and the technique has been in use for a long time. But, I’m not a compositor and this isn’t a compositing Patreon! What can we do with deep as 3D artists?

The big use case is depth of field. Before continuing, let’s take a look at what apps we can use for deep.

All production renderers support deep, including Arnold, RenderMan, V-Ray and Karma. Cycles does not.

On the compositing side, Nuke has had deep for years and has first class support for it. Fusion recently added deep tools, but they’re available only in the commercial edition of Resolve/Fusion. After Effects does not support deep.

Nuke offers a full featured non-commercial edition, but output resolution is limited. This might be fine for small personal projects, but many will want to upgrade to Nuke Indie for the best experience. It’s not unreasonably expensive (only moderately!) and of course you get all of the other Nuke stuff so you could make it your main compositing/image grading/video editing package if you wanted.

For the topic we’re going to discuss, depth of field using deep, you pretty much have to use Nuke for its Bokeh node (formerly pgBokeh). This is a powerful and feature-rich tool that has been in development for many years and is best in class. There are plenty of other Z depth-oriented DoF tools and plugins for other compositors, but if you want to use deep (and we do!), then Nuke’s Bokeh is it.

All that being said, let’s have a look at how this all works in practice.

I rendered a scene using RenderMan in Gaffer with deep output. This is very easy to set up: just add a deepexr output and that’s it. The workflow will be similar no matter what DCC or renderer you’re using. The EXRs are attached at the end of this article.

I denoised the beauty pass with the RenderMan ML denoiser. Note that in RMan it isn’t currently possible to denoise deep renders. I’m not sure it would matter much anyway, but for safety you could output a separate deep render with much higher samples. In this simple scene I didn’t notice much difference. The edge pixels move around a bit at different sample levels, but not enough to affect the final result. Pixar has suggested using a Deep Recolor operation to combine denoised beauty with high-sample deep layers.

Moving on, with our denoised beauty and deep EXRs ready to go, let’s open up Nuke.

Drag and drop the two images into the node graph. Beauty is looking beauty-ful and deep is looking… black.

Select the deep layer and now it’s looking yellow!

Much like Z AOVs, deep stores very large numbers in each pixel that are not immediately visible in the 0-1 range of our screens. Decrease the gain and now our objects are visible.

Compare this with the depth AOV in the beauty render (represented in red instead of yellow) and they look rather similar.

Notice the fringe around the edges of every object in the deep image. That’s the deep magic at work, storing transparency information for the anti-aliased edges.

For a fun visualization of what’s going on, connect a Deep Recolor, a camera, and a Deep to Points node:

Deep Recolor merges the deep and beauty images into one so that our pixel positions now have the color data from the beauty layer. We can see a 3D representation of our deep image, where every pixel occupies a spot in 3D space.

Note that completely occluded objects have no data, for example, the background behind the dice. Any pixel with transparency could occupy multiple 3D positions.

Let’s put the visualization aside and set up our depth of field. Attach a Bokeh node and connect it as follows. (I renamed the node to BokehDeep.)

There are a lot of settings to play with. I enabled Real World Lens Simulation for extra realism. One of the most important parameters is the Focal Plane, which sets what part of the image is in focus.

You can look at your deep layer to find the position of the object you want to focus on. Mouse over a pixel that should be in an in-focus area and copy its value over to the Bokeh tool’s Focal Plane parameter. Here the front corner of the purple dice is at about 3.8.

Adjust settings until you’re happy and now our render has cool and realistic depth of field, including aperture blades and chromatic aberration. Looks great!

What would happen if we used the same settings but in Z depth mode? The Bokeh tool supports that too. Well, let’s see. I copied the node and disconnected the deep input.

It looks okay…

Right…?

Here’s the same area using deep:

This is the risk you run with any type of post processing using Z depth. Edges, transparency and overlap can degrade significantly. You might get lucky in simple scenes and get a useable result, but more often than not there will be artifacts that are impossible to eliminate. To be fair, I didn’t spend any time wrestling with this. It might be possible to spruce up the Z version a bit, but why bother when deep worked perfectly out of the box? In more complex scenes it would be no contest, especially with difficult objects like hair.

Conclusion

There you have it! A quick but thorough introduction to how deep images work and how to use them to create beautiful depth of field.

The only downside to deep is that the file sizes are larger. If you’ve got a long animation you’re going to be eating up more disk space. Generally speaking, rendering in-camera depth of field will increase render times a lot. Post-processing using deep in Nuke, while an extra few steps, will save time when many images are involved, plus it’s much more flexible. Check it out and have fun!