Basic Lip-Sync Theory
Be sure that your new dialogue is in sync with your Source Media.
What to look for
Does the vub fit into the larger story you are trying to tell?
Does the output look like natural speaking?
Do you BELIEVE the character is saying the dialogue?
Is the output in sync?
Are certain mouth movements missing?
Do the lips close and open on the correct sounds?
đź‘„ Keep In Mind
Use the Source media as much as possible.
When a character’s mouth movement does not need to be changed from the original performance bring the Interpolation to 0
Does the transition from 100 to 0 Interpolation look good?
Adjust the time it takes to go from 100 to 0 to make the transition look as natural as possible. The minimum recommended movement is 4 frames.
Check for things that can’t be happening
A good foundation of any Vub is to remove any physical impossibilities. Here are some of the most common factors that can make or break your Vub.
B, M, P
M: closed during sound production, if the lips are not together the 'm' cannot be heard
B & P: closed right before sound production, the lips have to be open to hear the start of the word
F, V
The lower teeth MUST NOT show.
There MUST NOT be a visible gap between the upper teeth and the lower lip.
The lips MUST NOT be fully closed.
The jaw MUST NOT be significantly open.
The jaw is typically NOT fully closed.
Jumps and snaps
Often in speech, the lips will have a natural/constant movement until motion in a different direction is applied. In performance transfer, sudden jerky movements will make things not feel right but it can be hard to spot when playing back. Keep the mouth shape consistent between frames.
Re-assess the original performance
Watch the "Source media" and pay attention to the original expressions. There are less rules to follow when judging an original performance against a vub but often in DeepEditor the new dialogue will be delivered with the same intensity as the original performance.
Compare key moments in both performances and assess if there are similarities. Two examples are:
-
When the mouth is at its most open
-
When the mouth is at its most funneled or puckered
The majority of these checks should take place during the PT session (having the slapcomp and the plate in the same player is a big help) but re-assessing the plate with the original audio will be beneficial.
False Mouth Movements
A false movement refers to a moment in the Vub where the mouth is moving, suggesting dialogue when it isn’t there.
Similar to re-assessing the original performance these checks should take place during the refine session, but these moments can be easier to spot when watching with fresh eyes after downloading.
There are two main causes of false mouth movements.
The first is interpolation ramps being mistimed and bringing in some lip movement from the original plate. Check your interpolation ramp is working as intended.
The second is an override that doesn’t move with the driving data. This sometimes needs a deep dive to fully explain and future versions of this document will have examples but the core idea is that overrides that move with the driving data tend to help refine a PT whereas overrides that move against driving data will cause false movements.