You have all the shape keys you need and have rigged them to your model. Now the question remains for how to create realistic ‘lip syncing’ motion for your model. In other words, how can you get the lips to move in a natural way while following audio dialogue imported into Blender. To get to here, start from the beginning to reduce confusion – https://youtube.com/playlist?list=PLwMc23U27eSGgLaFemm6i8HKScNzpJste
In the last tutorial we looked at how we can rig facial expressions, so that shape keys can be affected with an external rig and the advantages.
This time we’re going to continue on from that and talk about how the lips can be properly animated to appear realistic in their motion. I can synchronize the audio of someone’s speech to the motion of the lips moving in Blender.
We talked already about the facial animation coding system, but what this allows you to do is have a systematic way of defining how the lips can move around, so that you can be sure that your shape keys are going to affect these lips in a more realistic way.
The next thing you’re going to want to do is have some kind of direct frame of reference of how this relates to speech patterns. The easiest way is to just make it yourself so get a camera, get a person: preferably someone who doesn’t really know why you’re doing this so they wouldn’t accidentally over-emphasize their lip motions.
Just just tell them you need them to say something, and you’re going to get them to say something called a phonetic paradigm.
Now I’m sure you’ve heard of the ‘quick brown fox jumps over the lazy dog’, a sentence that contains every letter of the alphabet. A phonetic paradigm contains every sound of the English alphabet.
Now what I specifically used as a phonetic paradigm doesn’t matter, and you have to source your own. But you want to get the person to look into the camera and just say the words. Now what you want to do is open that in a the Video editor if you need to.
You’re going to want to go through the video and find every single distinct sound. Go through all of them and take a snapshot an image of what the lips look like at that specific sound.
When you’ve compiled all of these images you can find the ones which look the most similar. They press both the lips together and then release them, and since these are so similar wee can call them the same thing.
Once you have a reference for every single phonetic sound, you can go about deleting the ones which aren’t necessary, since they can be filled by the facial shapes of other sounds. Once you’ve got your list of phonemes and the facial shapes they refer to, you’re going to want to open up all of your reference images, and try and replicate each and every one, by modifying the shape keys with the rig you’ve already made, to try and replicate how it looks inside of your reference image.
Now for everyone that you’ve made, you’re going to want to select the shape keys, whichever shape keys you have for controlling the lips and the mouth. Go over to the Pose library add a new Pose library, call it phonemes or lips or speech.
And press the add button and add new, naming it properly for whatever sound you have. Once you’ve got all of the ones you need, depends on how exact you want to be, and which ones you feel like you need, you should have a list of all the different sounds that you need to have your proper speech.
Once you’ve got your list of phonemes down, you can make a test. The easiest thing to test is to use the audio from the recording of the phonetic paradigm that you used initially to play it side by side with your 3d character.
Let’s say that you have some speech. Clear it so there’s no interference in terms of animation. Default everything and go to the first instance that the sound starts, so when you hear the sound go and apply the shape key.
Now it’s going to move to a third sound. You get the idea, and then a third again. And then the mouth closes. If you time this right with your audio, which I’ll get to sort of show you quickly, you can go to Object Mode and add a speaker, and put in the sound to here, and you’ll be able to hear audio at the same time as when you play the clip, and you’ll be able to hear the audio, and judge if it’s accurately timed.
That’s really it. That’s all you need to do to get shape keys to rig the face to animate the face, and then finally to get your lip syncing working and get realistic character motion. Now of course combine this lip motion with other things such as the eyebrows moving and you can get a pretty realistic result.
I think that’s everything we have for this time, adding it to your animation it just gives it a sense of added realism, which shouldn’t be overlooked.
