Facial Expression Control That Feels Emotional! (Patreon)

Hello, everyone!

I’ve been working on facial expression control lately, and it’s coming together nicely, so I wanted to share it with you.

What I was aiming for was expression control that conveys emotion. Instead of switching expressions with a button or key press, I wanted the face to change naturally along with facial tracking, and that’s what I really cared about.

It doesn’t just move the parts to match the tracked expression. I also added a control called an “emotion score,” which changes the face to match the emotion being expressed. For example, for a sad face, it checks things like whether the inner eyebrows are raised while the outer ends are lowered (a worried, slanted-eyebrow shape) and whether the corners of the mouth are turned down, and uses those as conditions for the expression.

If you only move the parts to follow the expression, the eyebrows and mouth just move around and you can’t really see any change in emotion. Adding emotion detection on top of that lets the face change into an expression that fits the feeling. The detection isn’t strict, though. I kept it fairly simple.

By the way, I also went ahead and implemented a control using ARKit, the so-called Perfect Sync, for now.

Since it captures the shape of the face very precisely, you can express emotion just by moving the expression parts to match. The catch is that there are a lot of BlendShapes. On top of that, if you want to tweak an expression, you have to go back to the BlendShape setup in the modeling stage, so it’s quite a lot of work. There are more handy tools these days, so I think it’s easier to make than it used to be, though.

Going back to the emotion score approach, you can set your own favorite expressions for each emotion, so I think it opens up fun ways to use it, like switching expressions depending on the situation. For example, when the face is smiling, closing the eyes can turn them into happy, curved smiling eyes.

I’m thinking of Perfect Sync as more of a “there for anyone who wants to use it” feature. The main one will be the emotion score.

For now, I’ve got the emotion-driven expression design in good shape, so next I’ll move on to swapping in characters and working on the UI design. Once those are in, I think the finished picture will start to come into view, and I’m really looking forward to it!