Everything You Ever Wanted to Know About Cryptomatte (But Were Afraid to Ask) (Patreon)

We’ve all used cryptomatte in our renders. It’s a convenient way to automatically create mattes (masks) for later use in compositing and editing. In the old days we’d have to manually create AOVs to ID specific objects in our scenes, a slow and potentially error-prone process. Cryptomatte was released about 10 years ago as an open source solution to this problem. Simply by adding cryptomatte AOVs to our renders we’re guaranteed to have a complete set of mattes for every asset, object and material. It’s easy and as close to set it and forget it as things get in 3D.

That said, we’ve also all probably wondered at some point: “Why are there 3 (or more) layers for each matte type, like cryptomatte_material00, 01, and 02? Why do some color channels contain such strange looking shapes and outlines? Why is there a red value of, uh, 18593.10204 in these pixels?”

Let’s learn how this all works!

Cryptomatte Theory

The general idea behind cryptomatte is that it automates the identification of elements from a 3D scene in a 2D render, either by asset (namespace), object or material.

Assets are collections of objects, objects are individual meshes, curves, etc. and materials are, well, materials.

For example, your 3D scene hierarchy might look something like this:fish01  eyes    mat_eye  body    mat_body_gold  fins    mat_finsfish02  eyes    mat_eye  body    mat_body_blue  fins    mat_fins

In that setup, fish01 and fish02 are assets, eyes, body and fins are objects, and materials are assigned to the objects.

One of the most user-friendly parts of cryptomatte is that it retains the original names of everything in the 3D scene. We can use the cryptomatte UI in Nuke, for example, to easily select fish01 and create a matte for all those pixels. But how are the names of our objects encoded in the image? We can’t call pixel data in an image file fish or /geo/spaceship/landing_gear. Numbers only!

Cryptomatte solves this by hashing the names of each object. This means using an algorithm to transform a word into a number. The name of every object in a scene is “hashed” into a hexadecimal number that can then be encoded as a pixel value.

Cryptomatte stores a manifest that tracks what names corresponds to which hashes.

For example, an object called “fish” in our DCC might be hashed as c5bad3d0 in the manifest and rendered as -5978.4609375. (For an advanced look into how cryptomatte can convert hashes into pixel values, check out the hex_to_float function here: https://github.com/Psyop/Cryptomatte/blob/master/fusion/Modules/Lua/cryptomatte_utilities.lua)

This is all well and good when creating very simple mattes, but what happens when two objects overlap in 3D space and occupy the same pixel in the render? Think of motion blur or anti-aliasing around an object’s edge. In those cases, objects may be partially visible behind each other.

To solve this problem, cryptomatte uses an algorithm to calculate the sampling contribution of each object to the final pixel and ranks the objects accordingly. One way of thinking of it is that Rank 1 contains the “most important,” foreground elements, Rank 2 contains parts of the scene that have been calculated to contribute fewer relative samples to the final pixel, and so on. Cryptomatte recommends a default of 6 ranks to cover the vast majority of cases. It’s a rare situation where more than 6 objects are visible in the same pixel.

Along with each rank, cryptomatte creates coverage maps which store the contribution of each Rank to the final pixel. If a pixel from Rank 2 is visible behind one from Rank 1, the coverage map will give us the percentage contribution of each rank.

So, to sum up: Cryptomatte takes asset, object and material names from a 3D scene and hashes them into numerical representations. The relationship between names and hashes is stored in a manifest. ID mattes for each object are automatically created using a ranking algorithm. The ranks are stored as channels in layers in an image file. Each rank has an associated coverage channel. 

Whew, that’s a lot! How does all this theory play out in practice? Let’s look at a render to see how it works.

Cryptomatte in Practice

Here’s a render of some dice created with Cycles in Gaffer.

I used Nuke to shuffle the external cryptomatte channels into the main image.

The resulting image consists of an RGBA “beauty” layer and three cryptomatte layers. This cryptomatte workflow is essentially the same in any compositing or image-editing app: Fusion, After Effects, Gaffer, you name it. The steps I take here in Nuke should be reproducible everywhere.

In this image there are only cryptomatte_object00, cryptomatte_object01, and cryptomatte_object02 layers. I didn’t need asset or material mattes so I didn’t create those layers.

What do the number suffixes in each layer name mean?

Remember that cryptomatte recommends 6 ranks total. We’ve got 4 available channels per layer, Red, Green, Blue and Alpha, and three layers, 00, 01, and 02. That’s 12 total channels. For each of our 6 desired ranks, each has a coverage channel associated with it. 6 ranks plus 6 coverage channels equals, yep, 12 channels!

Let’s look at cryptomatte_object00. We know this contains 4 channels and thus it holds the data for Ranks 1 and 2 and their coverages. Press R in Nuke to view the red channel. This looks like a pretty much perfect matte of our dice and background.

Mouse over the pixels and look at the red channel.

Notice how each object has its own red value. These are the converted float values associated with our hashed names. We can see the manifest contents in the EXR metadata.

It doesn’t make any difference what these “colors” are. The only thing that matters is that the numbers are unique and associated with the correct manifest entry.

Now, press G to view the green channel. This looks like an outline of the dice.

Flip back to the original image. Note that the edges of each dice have anti-aliasing blur. We see part of the foreground dice and part of the backdrop behind it.

This is the purpose of the G channel: it sets the coverage of the Rank 1 objects. Where any dice is fully visible, the G channel has a value of 1, meaning it contributes 100% of the pixel value. Mouse over the anti-aliased edges of the dice. We can see that our R value says this is dice02 (-5978.4609375), and the G value assigns it 80.804% coverage.

Look at the beauty layer and indeed, the dice in this pixel is partially transparent, just as the coverage channel indicates.

Now, view the B channel. This is the Rank 2 channel. This channel identifies which objects contribute fewer samples to a given pixel compared to Rank 1.

As might be expected, coverage for Rank 2 is contained in the A channel. Let’s look at the same pixel we did for Rank 1.

Here we see the float value for the /set/backdrop object is 0.37232 (the B channel) and its coverage in this pixel is 19.196%. What was the coverage of the Rank 1 dice pixel in this location? 80.804%. And what’s 80.804 + 19.196? 100%!

Cryptomatte has successfully accounted for the Rank 1 dice pixel and the Rank 2 backdrop pixel slightly visible behind it through the anti-aliased edge. The contributions of those two objects add up to 100% of the final pixel value.

Note that everything aside from Rank 1 is only a very small part of the scene. Rank 1 is the majority of what appears in this image: the foreground objects. Rank 2 contains only the anti-aliased edges around each dice. There are zero Rank 2 pixels in, say, the center of dice01 because it is the only object visible in that rank. This same methodology holds true for Ranks 3 and beyond.

Conclusion

And there you have it, a crash course in cryptomatte! It’s a brilliant system that is at once very complex and sophisticated and also extremely easy for artists to use. As always, it helps us do better work when we understand our tools, and now we understand very well how cryptomatte works under the hood!