I Tried Making All Kinds of Lip Sync (Patreon)

Hello, everyone!

I've been working on character control lately and tried out a bunch of different approaches to lip sync, so I wanted to share them with you. Things are starting to come together nicely.

Anime-Style Lip Sync

First up is anime-style lip sync. Like the mouth movements you see in anime, this switches between closed, half-open, and open mouth shapes to create the motion.

Instead of just cycling through the shapes in order, I put a lot of thought into the switching itself, aiming for that sharp, snappy timing you get in anime, with clear holds between switches.

Vowel-Based Lip Sync

Next is vowel-based lip sync. This is the kind that syncs to vowel sounds, basically the "a, i, u, e, o" of Japanese. For the voice input, I used uLipSync, developed by hecomi.

If you just switch shapes directly based on the voice input, the movement ends up feeling soft and mushy, so I worked out a way to add more contrast and snappier timing to it.

Hybrid Version: Face Tracking × Voice

Last up is a hybrid version that combines face tracking with voice input.

Face tracking picks up the facial expressions, and the voice input handles the mouth movement, so you get the best of both. It also has voice-based mouth movement built in, so the mouth still moves even when facial expression changes are subtle. Simply combining the two as is causes the expression values to get too large and things start to look off, so I put some work into fine-tuning that balance.

I'm planning to write a more technical breakdown of all this in a separate post down the road. Putting together a proper technical write-up takes a fair amount of effort, so for now, consider this the quick introduction!