How to Achieve Specific Mouth Shapes

When it comes to lip sync, not all phonemes were created equal - some have more stringent requirements compared to others.

The highest priority lip shapes to get right are the shapes for:

• p/b/m
• f/v
• w/beginning syllable r/oo

The lower priority lip shapes are those for:

• th
• s
• z
• ch/sh/dge/zh

Highest Priority Lip Shapes

p/b/m

p: 

General Principles: Ensure the lips are closed in the moments leading up to the p’s sound burst. Look at the waveform for clues; the waveform for p should feature a period of silence followed by a burst.




Applying the Principles: To ensure the lips are closed in the moments leading up to the plosive burst, there must be no visible gap between the top and bottom lips. (Remember–no rogue openings!) The top and bottom lips must press together or roll into the mouth.

When refining vubs with DeepEditor, the following scenarios and key blendshapes will help you achieve successful syncs for p’s! 

• If the jaw is not visibly or significantly open for the previous sound → negative mouthFunnel is recommended: 
  • Negative mouthFunnel rolls both lips into the mouth and brings them together. 
  • We recommend setting negative mouthFunnel to -10 to -12 
    • Beyond this range, negative mouthFunnel can look unnatural/uncanny.
    • Be conservative but keep in mind that conditions do vary.
• If the jaw is visibly and significantly open for the previous sound → negative jawOpen is recommended.
  • Negative jawOpen brings both lips closer to each other by bringing the lower jaw up.
  • We recommend setting negative jawOpen to -10; however, conditions may vary. There may be rare cases where more extreme measures are needed e.g. -30 jawOpen may be appropriate. Extreme ranges like this are reserved for counteracting very open jaws.
• If there is a rounding sound before/after m/b/p (e.g. w/r/oo) → the following activations are recommended:
  • mouthFunnel ~ +10-20 
  • mouthPucker ~ +10-20
  • NOTE: Employing a positive mouthFunnel and/or pucker is only possible if you are able to achieve a proper closed lip shape from negative jawOpen or other combinations. You cannot activate a negative mouthFunnel with a positive mouthFunnel or positive mouthPucker.
    
    

b:
General Principles: Just as with p, ensure the lips are closed in the moments leading up to the b’s sound burst. However, because b is voiced and p is typically unvoiced in American English, the b waveform will look less distinct than the p waveform–i.e. when looking at the waveform for b, instead of an obvious silent moment, you will see smaller, less pronounced waveforms preceding the release.


Applying the Principles: To ensure the lips are closed in the moments leading up to the plosive burst, there must be no visible gap between the top and bottom lips. (Remember–no rogue openings!) The top and bottom lips must press together or roll into the mouth.

When refining vubs with DeepEditor, the following scenarios and key blendshapes will help you achieve successful syncs for b’s! 

• If the jaw is not visibly or significantly open for the previous sound → negative mouthFunnel is recommended: 
  • Negative mouthFunnel rolls both lips into the mouth and brings them together. 
  • We recommend setting negative mouthFunnel to -10 to -12 
    1. Beyond this range, negative mouthFunnel can look unnatural/uncanny.
    2. Be conservative but keep in mind that conditions do vary.
• If the jaw is visibly and significantly open for the previous sound → negative jawOpen is recommended.
  • Negative jawOpen brings both lips closer to each other by bringing the lower jaw up.
  • We recommend setting negative jawOpen to -10; however, conditions may vary. There may be rare cases where more extreme measures are needed e.g. -30 jawOpen may be appropriate. Extreme ranges like this are reserved for counteracting very open jaws.
• If there is a rounding sound before/after m/b/p (e.g. w/r/oo) → the following activations are recommended:
  • mouthFunnel ~ +10-20 
  • mouthPucker ~ +10-20
  • NOTE: Employing a positive mouthFunnel and/or pucker is only possible if you are able to achieve a proper closed lip shape from negative jawOpen or other combinations. You cannot activate a negative mouthFunnel with a positive mouthFunnel or positive mouthPucker.
    
    

m:
General Principles: Unlike p’s & b’s, m’s do not feature spikes. M’s are voiced and take place only during lip closure. For this reason, ensure that the lips are closed during m’s. Due to their lack of abrupt energy releases, you can find m’s by their smooth and consistent waves free of jarring spikes. 

Applying the Principles: To ensure the lips are closed during the entirety of the m production, there must be no visible gap between the top and bottom lips. (Remember–no rogue openings!) The top and bottom lips must press together or roll into the mouth.

When refining vubs with DeepEditor, the following scenarios and key blendshapes will help you achieve successful syncs for m’s! 

• If the jaw is not visibly or significantly open for the previous sound → negative mouthFunnel is recommended: 
  • Negative mouthFunnel rolls both lips into the mouth and brings them together. 
  • We recommend setting negative mouthFunnel to -10 to -12 
    • Beyond this range, negative mouthFunnel can look unnatural/uncanny.
    • Be conservative but keep in mind that conditions do vary.
• If the jaw is visibly and significantly open for the previous sound → negative jawOpen is recommended.
  • Negative jawOpen brings both lips closer to each other by bringing the lower jaw up.
  • We recommend setting negative jawOpen to -10; however, conditions may vary. There may be rare cases where more extreme measures are needed e.g. -30 jawOpen may be appropriate. Extreme ranges like this are reserved for counteracting very open jaws.
• If there is a rounding sound before/after m/b/p (e.g. w/r/oo) → the following activations are recommended:
  • mouthFunnel ~ +10-20 
  • mouthPucker ~ +10-20
  • NOTE: Employing a positive mouthFunnel and/or pucker is only possible if you are able to achieve a proper closed lip shape from negative jawOpen or other combinations. You cannot activate a negative mouthFunnel with a positive mouthFunnel or positive mouthPucker.
    

f/v

f:
General Principles: Ensure that the lower lip is covering the lower teeth and there is no visible gap between the upper and lower teeth. Because f is an unvoiced fricative, its  waves tend to be more distinct and chaotic than other sounds. An f waveform can be more easily identified than a v, which is voiced and has a less distinct set of waves.

Applying the Principles: To ensure the f lip shape is readable, there must be no visible gap between the upper teeth and lower lip. (Remember–no gap errors!)

When refining vubs with DeepEditor, key blendshapes will help you achieve successful syncs for f’s:

• negative mouthFunnel set to - 10 + positive mouthShrugUpper set to +10-15
  • Negative mouthFunnel is used to bring the lower lip inward. Because mouthFunnel does not have an upper/lower split, we also need to add mouthShrugUpper to counteract the rolling of the top lip. 

• In extreme and rare cases where extra articulation is needed, a +10 jawForward may be helpful.

v:

General Principles: Ensure that the lower lip is covering the lower teeth and that there is no visible gap between the upper and lower teeth. Though v can be harder to identify than f in terms of its waves, you can often identify v by looking for a slightly noisy set of waves with a consistent amplitude. In the example below, you can see that the v waves are consistent and clearly different from the taller “ie” vowel waves. You can also see a clear change in wave height appear as the subsequent word (“you”) takes place.

Applying the Principles: To ensure the v lip shape is readable, there must be no visible gap between the upper teeth and lower lip. (Remember–no gap errors!)

When refining vubs with DeepEditor, key blendshapes will help you achieve successful syncs for v’s:

• negative mouthFunnel set to - 10 + positive mouthShrugUpper set to +10-15
  • Negative mouthFunnel is used to bring the lower lip inward. Because mouthFunnel does not have an upper/lower split, we also need to add mouthShrugUpper to counteract the rolling of the top lip. 

• In extreme and rare cases where extra articulation is needed, a +10 jawForward may be helpful.

w/r/oo

• w/r/oo:
  AKA → /w/, /ɹ/, & /u/
  Because w, r, oo are not highly stable shapes, you will primarily want to use them to contrast against the shapes around them. Compared to a lip-widening sound like “ee,” the w/r/oo should showcase a narrower mouth shape. Compared to a very open sound like a h or an ah, the mouth should be more closed.

General Principles: Ensure the lips are rounded with a minimal opening. The lip opening can vary somewhat in size depending how tensed the lips are; however, the opening tends to remain quite small. At the smallest level, it can be difficult to even see that the lips have an opening.

Applying the Principles: To ensure the w/r/oo lip shapes are readable, it is recommended to encourage rounded shapes–or at least contrastive shapes that are more rounded compared to surrounding shapes.

When refining vubs with DeepEditor, the following scenarios and key blendshapes will help you achieve successful syncs for w’s, beginning r’s, and oo’s: 

• Under general conditions use:
  • mouthPucker at a range between + 8 - 20
• Specifically for oo, you may also consider adding:
  • mouthFunnel (in addition to mouthPucker) at a range between + 8-20. 
• If you are reviewing a character with a side profile view:
  • You may need to add a slight mouthFunnel.
• If the starting position features  jaw and/or lips that are too open, the following activations are recommended:

• negative jawOpen at -10
• negative mouthShrugUpper between negative -10-15

Lower Priority Lip Shapes

th

th:
AKA →  /θ/ & /ð/

General Principles: The main feature of the th sound is that the tongue slips between the teeth. To make this feature believable, ensure the jaw is not fully closed or too closed to allow for a hypothetical tongue protrusion.

Applying the Principles: Though the tongue is not currently a controllable feature, the distance between the upper and lower teeth has to be believable for a perceived tongue interaction.

When refining vubs with DeepEditor, the following key blendshape recommendation will help you achieve successful syncs for th: 

• Keep the jaw slightly open but open no more than a (flat) pinky distance between the upper and lower teeth.

s

General Principles: The main requirements for the s sound are that the lips are parted and the jaw is either closed or nearly closed. The close proximity of the upper and lower teeth is what makes the airy, fricative sound of s. Like p and f, s is also unvoiced and takes on a distinct, chaotic pattern. Its waves are quite compacted without a clear, repeating pattern.

Applying the Principles: Ensure the lips are open and the teeth are touching or nearly touching. You may achieve the appearance of teeth closure by ensuring the jaw is closed or almost closed, i.e. jawOpen is set near 0 (at low positive or low negative value).

When refining vubs with DeepEditor, the following key blendshape recommendation will help you achieve successful syncs for s: 

• Keep jawOpen closed or slightly open; the jaw for s sounds must be more closed compared to th sounds.
• You may trick the renderer into giving you the teeth you need by correcting the placement of the mouth; this can be done by setting mouthShrugUpper and mouthShrugLower to the same value.

z

General Principles: Like the s, the main requirements for the z sound are that the lips are parted and the jaw is either closed or nearly closed. While the waveform for z is less distinct than s due to being voiced, the rest of the criteria for s is the identical.

Applying the Principles: Ensure the lips are open and the teeth are touching or nearly touching. You may achieve the appearance of teeth closure by ensuring the jaw is closed or almost closed, i.e. jawOpen is set near 0 (at low positive or low negative value).

When refining vubs with DeepEditor, the following key blendshape recommendation will help you achieve successful syncs for z: 

• Keep jawOpen closed or slightly open; the jaw for s sounds must be more closed compared to th sounds.
• You may trick the renderer into giving you the teeth you need by correcting the placement of the mouth; this can be done by setting mouthShrugUpper and mouthShrugLower to the same value.

ch/sh/dge/zh

ch/sh/dge/zh:
AKA → /tʃ/, /ʃ/, /dʒ/, & /ʒ/

General Principles: Like s/z, the ch/sh/dge/zh sounds also require open lips and touching or nearly-touching teeth. In many cases you may notice the lips funnel or protrude outward.

Applying the Principles: Because funneling/protrusion is not a reliable criteria for ch/sh/dge/zh, you will need to focus on jaw openness.. If you see a significantly open jaw (i.e. notable distance between upper and lower teeth), that is a sign that the audio and video are not in sync or your shape is not believable.

When refining vubs with DeepEditor, the following key blendshape recommendation will help you achieve successful syncs for ch/sh/dge/zh:

• In general, ch/sh/dge/zh typically requires:
  • negative jawOpen set to -10
  • positive mouthFunnel as low as +10  as high as+ 30
• If you need more upper teeth to show add positive mouthShrugUpper set to +10

Other vowel movements

Though vowels don’t typically have as reliable features as consonants, they will generally require more open jaw and lip positions. You can use vowels to compare against the consonants around them. We will have more documentation on vowels in the near future.

Images made with content from from:
Title: Quick Fall Makeup Tutorial + My Mini Sephora Haul
Author: C Key
Source: https://www.youtube.com/watch?v=kCjvDgK4oTg
License: Creative Commons Attribution license (reuse allowed)