Stencil Buffer Shaders - Experiments & Thoughts (Patreon)

With Godot 4.5 here, I've been experimenting and trying out stencil shaders and looking everywhere for new effects and tricks to use them with.

For those who don't know, the stencil buffer is simply a black and white image with 255 values. You can use your meshes to write to this buffer. And create meshes that only render if the pass a test. Usually something like: "Only render parts of this mesh where the stencil buffer is 1"

Essentially, the stencil buffer is an easy way to mask meshes at very low cost. And there are quite a few neat little tricks you can do with this.

Experiments

The classic stencil effect of outlines and xrays are already built-in into Godot's material now, so let's not talk about that.

So let's start with the classic "Impossible Geometry" or "Portal effect". Where two meshes seems to occupy the same space and are only visible when you look at them at a certain face of a cube. Or the portal effect where the object is only visible from one side.

And using this same principle, I've make an x-ray scanner.

The character here have an invisible bone behind that are only rendered on the scanner mesh.

There are two possible ways to achieve this.

Making the bones to write to the stencil 1, and then have the scanner be a solid mesh that only renders on stencil 1 zones. ( the blue skeleton on the right )

Or making the bones always visible by disabling the depth test, but masking them so they only render on stencil 1. And then have the scanner area write to stencil 1.

(Red is stencil 1 and yellow is stencil 2. I used different ones so both effects don't interfere with each other)

Post-Effects

Another useful effect is custom per-object post-effect.

These are very useful, you can mask only certain objects to receive the shader effect by combining it with the SCREEN_TEXTURE. Here's a desaturation shader that is only affecting the two front objects.

You can do scanner effects to highlight objects, blur, pixelate and distort effects.

Other effects

I've been looking around what possible effects there are but its been hard.

Almost everywhere I looked are the same effects: Some variation of Portals or Impossiblle geometry. No more complex tricks anywhere. (And I know the exist)

One that I found is the appearing portal effect.

Which is a bit more complex that you might initially think.

It uses 4 meshes to create a volume so that the characters can show its arms on top of the portal as well as on the bottom through the portal.

And after the portal is done its animations, I need to call a functions to remove the stencil so the character can render normally.

---

Another cool stencil trick I found was fake stencil lights.

This is a hard one. And I only found a single unity tutorial about it. And the demo inside Godot's stencil PR that I just happened to follow.

---

I also found a cool video about replicating Split Fiction in unity (https://www.youtube.com/watch?v=PgCiQHmZFBE)And while this is a variation of the impossible geometry effect, the final video result is very nice.

Compositor effects

While not very easy. It is possible to use stencil inside compositor effects for full screen post-effects. It is not officially supported, but you can create your custom stencil copy render pipeline.

Compositor effects use compute shaders and pure glsl, so they are already hard to use compared to gdshaders.

You have to create what is essentially a new render pipeline to copy the stencil.

Its a bit complicated but you can create a RDPipelineDepthStencilState.new() to set the stencil parameters like in the materials, and use that when creating a new render pipeline with RenderingServer.get_rendering_device().render_pipeline_create()

I don't really recommend it though.

Impossible geometry problem

Now, the one BIG problem that impossible geometry have are shadows.

I didn't found a simple example of masking shadows in either Godot, Unity or Unreal in the examples that I researched.

Here's what happens. If you use shadows, they aren't masked. So you'll see shadows even if your object is not there.

Which in a real-game scenario is not very good. Seeing floating shadows for objects that aren't there feels very buggy.

So what everybody does, is just not use shadows inside stencil objects. Which is a bummer.

The single example with working shadows in the whole internet that I found is Godot's feature video. That has shadows inside, while not on the outside.

I'm not sure how that is possible. I've tried a few things, including trying to create a fake ground to 'catch' the shadows, or use camera-based shadows. None of the things I tried worked.

If we had access to the current stencil id inside a shader, I think I could do something by using a custom shadow pass. But even that would not fully solve the problem. When the object starts appearing through the stencil, the shadow would start showing up too, and depending of the casting angle, appear behind the door/portal.But it would be an improvement at least.

So I think the Godot example is using some sort of trick to fake the shadows. Either they are baked in, of there is a second camera sub-viewport casting the shadows. I tried asking passivestar on their bluesky post, but the post in a bit old by now and I got no answer there. So it remains a mystery on how that is possible or what tricks were used there.

What it still lacks

Godot Stencil support is not fully there yet. Some more advanced operations are not available and we lack the ability to query the stencil id inside the shader. Even a basic stencil_id variable to sample the stencil inside the shader would allow for much more advanced types of shaders.

Right now, you can't have one single shader do multiple things depending on the stencil id. You can do some simpler multi-stencil effects, but even then, you'd need to create multiple render passes with multiple variations of the shader for each stencil id effect. Not Ideal.

Another thing is that it is not very fun having to modify your materials to add stencils if you have many objects that need them.

Unity's solution to this is to have stencils as a render feature and that you can mask objects just by adding them added to custom layers, without changing the materials.

This solution is great, because it also allows you to dynamically change the stencil through code by simply changing the layers of the objects.

Godot can change stencils with code, but only if you use the standard material because it re-generates the shader code dynamically at runtime.

If you created your own custom shader, you can't change the stencil_mode. Just like you can't change the render_mode. So you need to create variations of that shader, which is very annoying . Maybe we can come up with a solution to dynamically alter the shader code like standard material does internally, but I'm not sure.

What's left to do

I learned a lot about stencils this week. I'll be on the lookout for more effects and tricks and keep a database of all sorts of effects that I can find.

I plan to do quick videos for each individual cool effect that I find so as to keep an online library of stencil effects. I'll try to keep the videos between 3-5 minutes and go strait to doing the effect.

So here's a tentative list for the future:

Per-object full screen shader effect

Scanner hidden geometry

Standard impossible geometry/Portal

Advanced portal appearing effect

Old-zelda style light effect

Show objects behind walls (mask walls when player in front)

Classic boat/water effect (Done)

End

I spent yesterday making assets to prepare the tutorial. Just to realize I'm very rusty with blender. It feels like I had to re-learn everything again how to weight paint and rig the character.

But that's what I've been doing.

I'll try to come up with a tutorial for one of the effects next.

See you next time.