How To Evaluate Lip Sync

Workflow Tips
How To Fix the Most Common Issues in a Vub
CORCTET Method for Evaluating Lip Sync
Evaluating m/b/p
Evaluating Lip Rounding: w/r & oo
Evaluating Lip-Tooth Interaction for f/v
Evaluating Nearly-Closed Teeth: s/z
Vowels Overview

Workflow Tips

Before diving into the details for how to evaluate lip sync, here are a few basic tips to keep in mind.

TIP 1 - Start at 1x speed.

Watch the shot at normal speed to see if you can identify what's right or wrong at a glance.
There will be specific moments where something looks off, but you can’t tell why. Make a note of these moments.

TIP 2 - Slow it down.

Speech happens so quickly. It is extremely difficult to pinpoint what went wrong/right without manipulation.
After performing the initial spot check, slow things down and check the moments that felt “off.” Determine which phonemes were present during those moments.
Analyse them to identify what works and what doesn't.
Even if nothing felt “off” at 1x, it is still a good idea to search for errors or moments that are not convincing. (We will go over what to look out for in the upcoming sections.)
Zoom in to get a better look at the mouth movements.

TIP 3 - Ride the waveform.

After determining the phonemes you want to evaluate, you will often need to take extra precautions to make sure you are judging the correct frame. To ensure you are in the correct spot, use clues from the waveform. This part is quite critical. If you are even a fraction of a second too early or too late, it will disrupt your whole inspection and your findings will be irrelevant.
While there is no exact formula for what a waveform should look like for a particular phoneme, there are distinguishable properties between vowels, fricatives, nasals, etc.

TIP 4 - Observe the relationship.

Analyzing the correct frame is important, but single frames won’t tell you the whole story. Since viseme shapes are relative and not absolute, observing the relationship between the target shape and the surrounding sounds is necessary.
For example, if you are looking at the word “gloss” and evaluating the “ss,” you will want to ensure that relative to the “o” in “gloss” AKA the [a] phoneme, the “ss” jaw position is more closed - or at least, not more open. Visualizing the transition from a more open position to a more closed position is important.

How To Fix the Most Common Issues in a Vub

Let’s break down some of the most common and obvious factors that affect the success of a vub.

Rogue Openings

A rogue opening occurs when the lips part during a moment they should be fully closed. Rogue openings can also be described as false positive lip separations, and they can easily disrupt the believability of a performance. During these sounds, even a small gap between the lips can throw off the realism of your lip sync; so be sure to prioritize removing rogue openings.

To avoid rogue openings, pay careful attention to /m/, /b/, or /p/ sounds:

/m/ requires full lip closure for the entire duration of the sound.
/b/ and /p/ require full lip closure at their onsets, before a burst of air is released.

Gap Errors

A gap error occurs when there is a false positive gap between articulators (lips, teeth, tongue, etc.) that require contact with each other. For example, in order to produce a legible /f/ or /v/ sound, the lower lip must make contact with the upper teeth. When this contact is missed, the /f/ or /v/ loses its readability and harms the performance.

Overdrops

An overdrop occurs when there is more jaw opening than what is considered feasible for a given sound. Many consonants, especially fricatives like /s/, /z/, /ʃ/, /ʒ/, etc. require the jaw to be nearly closed. When the jaw is too open, it prevents the articulators from achieving frication (turbulent and noisy airflow produced when air is forced through a narrow passage).

Underdrops

An underdrop occurs when there is less jaw opening than what is feasible for a given sound. Vowels typically require more jaw openness compared to consonants. When the jaw is too closed for a vowel it can harm the readability of that vowel; it can also harm the readability of the surrounding consonants by creating improper contrast in the greater context.

Term	What it means	When it matters
rogue opening	when there is a false positive opening between the lips	during p’s, b’s, and m’s
gap error	when there is missed physical contact between key articulators	during f’s and v’s
overdrop	when the jaw is more open than is likely	during most consonants
underdrop	when the jaw is more closed than likely	during vowels

CORCTET Method for Evaluating Lip Sync

The CORCTET method is a set of practical strategies for evaluating lip sync. It allows you to examine mouth movements based on the following attributes:

CLOSED
OPEN
ROUNDED
CORNERS
TEETH
ENVIRONMENT
TONGUE

Use the questions under each attribute to see exactly what needs fixing in your vub:

CLOSED

Are the lips closed when they should be?
Cases when the lips should be closed include:
- during "m" production
- just before the sound from a "p" plosive
- just before the sound from a "b" plosive
If a phoneme other than "m", "b", or "p" is present, the lips should NOT be closed.

OPEN

Are the lips open when they should be?
Just as important as ensuring the lips are closed for "m", "b", and "p", we need to ensure the lips are open for all other non-m/b/p’s.

ROUNDED

Are the lips rounding for the appropriate phonemes (w, r, oo) or for phonemes that generally employ lip stretching/pinching (eh, ih, ee)?

CORNERS

Are the corners stretching or pinching for appropriate phonemes (eh, ih, ee, also s/z works) or for phonemes that generally require lip rounding (w, r, oo)?

TEETH

f/v- Are the top or bottom teeth showing?
th - Are the teeth closed or open? If open, how vertically separated are they?
s/z & ch/sh/dge/ʒ - Are the teeth touching/nearly touching or greatly vertically separated?

ENVIRONMENT

coarticulation - Are the lips rounding for an m/b/p? If so, does the following sound require lip rounding? If not, this may be inefficient! Efficiency influences how natural lipsync may or may not look.
relativity - s/z → a (“fall”) vs. a → s/z jaw opening
yelling/whispering/something in mouth
anatomy
accent/enunciation
emotion
original dialogue

TONGUE

th - Is the tongue showing between the teeth or is there a visible emptiness between the teeth?
n/l or a (as in “fall”) - Is the tongue raised upward, protruding, or laying flat?

Evaluating m/b/p

Phoneme	Graphemes	Word Examples	Voiced?
b	b, bb	bug, rubble	yes
m	m, mm, mb, mn, lm	man, summer, comb, column, palm	yes
p	p, pp	pin, dippy	no

As a recap, m/b/p is characterized by the meeting of the top and bottom lips to form a mouth closure.

m/b/p Rules

Regardless of various contexts like emotional state, coarticulation, speech rate, volume, etc. - there is one thing that must generally be true for m/b/p representation:

The lips must be closed. In general, the lips should be fully or almost fully closed for m/b/p sounds.

m - can produce continuous sound while the lips remain closed.
b and p - require a two-state process: closed to open. The lips close first, then the sound is released during the opening.

Evaluating Lip Rounding: w/r & oo

Phoneme	Graphemes	Word Examples	Voiced?
w	w, wh, u, o	wit, why, quick, choir NOTE: “Quick” is phonetically written as: kwɪk and “choir” is phonetically written as: ˈkwaɪəɹ	yes
r	r, rr wr, rh	run, carrot, wrench, rhyme	yes
u: (AKA “oo”)	o, oo, ew, ue, u_e, oe, ough, ui, oeu, ou NOTE: The _ in “u_e” indicates that there is another letter in between the u & e -e.g. flute.	who, loon, dew, blue, flute, shoe, through, fruit, manoeuvre, group	yes

At a basic level, w/r/oo is characterized by lip rounding with a small lip opening. The lip opening can vary somewhat in size depending on how tensed the lips are; however, the opening tends to remain quite small. At the smallest level, it can be difficult to even see that the lips have an opening. The jaw must be open to some degree.

w/r/oo Rules

Regardless of various contexts like emotional state, coarticulation, speech rate, volume, etc. - there is one thing that must generally be true for w/r/oo representation:

The lips MUST NOT be fully closed. Fully closed lips makes w/r/oo production impossible. (Air needs to be able to escape!) However, there will be cases where the lip opening is so slight that the lips appear closed. As long as it is not 100% clear that the lips are fully sealed, ambiguous opening can be acceptable.
The lips are most natural when rounded. Though the degree of rounding is variable, relative to other sounds (e.g. eh, ih, ee), w/r/oo tends to draw the lip corners toward the midline of the face and the upper and lower lips toward each other.
The lips MUST NOT be significantly open (vertically + horizontally). The w/r/oo shape is marked by its constrictive nature. If the jaw and lip openings are too wide and/or tall, then the shape will not be readable as w/r/oo.

An important note about r:

It really matters what position the r is taking in a word.
When r occurs at the beginning of a word - e.g. “rest” or “red,” the r is significantly rounded.
When r occurs at the end of a word or syllable - e.g. “mother” or “father,” the r is much more relaxed and undistinguished.

Evaluating Lip-Tooth Interaction for f/v

Phoneme	Graphemes	Word Examples	Voiced?
f	f, ff, ph, gh, lf, ft	fat, cliff, phone, enough, half, often	no
v	v, f, ph, ve	vine, of, stephen, five	yes

In general, f/v sounds require a small mouth opening to create a tight airflow. This is typically caused by the lower lip meeting the upper teeth. Sometimes you might not see the teeth easily due to factors such as lip size, enunciation, whether the teeth are pressing against the inner lip (inside the mouth) or the visible lip, etc. The jaw generally needs to be open to some degree in order to allow the teeth to separate and the lower lip to make contact with the upper teeth.

f/v Rules

Regardless of various contexts like emotional state, coarticulation, speech rate, volume, etc. - there are a few things that must not be true for f/v representation:

The lower teeth MUST NOT show. Due to the interaction of the lower lip with the upper teeth, lower teeth visibility is highly unlikely. If the lower teeth show, it will cause confusion in readability.

NOTE: In the extremely unlikely scenario that the lip-to-teeth meet is reversed, i.e., the upper lip touches the bottom teeth, the lower teeth show is possible. Though it is possible, it is best to assume this will never happen unless there is a rare case like missing upper teeth.

There MUST NOT be a visible gap between the upper teeth and the lower lip. If a gap exists between the upper teeth and the bottom lip, this means there is not enough constriction and contact to create a fricative sound.

The lips MUST NOT be fully closed. Fully closed lips make f/v production impossible. (Air needs to be able to escape!) However, there will be cases where the lip opening is so slight that the lips appear closed. As long as it is not 100% clear that the lips are fully sealed, an ambiguous opening can be acceptable.
The jaw MUST NOT be significantly open. Though it is difficult to draw the line at what might be considered “significantly open” (especially due to individual differences), this point is still necessary to keep in mind. The more the jaw opens past the minimum teeth separation point (~tongue-tip distance), the more difficult it is to produce a recognizable f/v.
The jaw is typically NOT fully closed. It is technically possible to produce f/v with a closed jaw, but a closed jaw will degrade the recognizability of f/v. Outside of clenched teeth, assume the jaw is slightly open.

NOTE: Due to lip occlusion, it is unlikely that we are able to visually determine whether the jaw is slightly open or fully closed for f/v’s; so, while this point is helpful to be aware of, in practice it is not likely to be an applicable metric.

Evaluating Nearly-Closed Teeth: s/z

Phoneme	Graphemes	Word Examples	Voiced?
s	s, ss, c, sc, ps, st, ce, se	sit, less, circle, scene, psycho, listen, pace, course	no
z	z, zz, s, ss, x, ze, se	zed, buzz, his, scissors, xylophone, craze	yes

The prototypical s/z shape involves slight tension in the lip corners (AKA dimpler), lip separation, and gently touching or gently separated teeth; however, the lip corner tension is variable and may or may not be present.

Due to the tongue placements for s/z, the more you separate your teeth, the more difficult it becomes to produce s/z in a recognizable manner. Because of this difficulty, teeth separation can be a reliable metric for assessing s/z believability.

s/z Rules Regardless of various contexts like emotional state, coarticulation, speech rate, volume, etc. - there are a few things that must not be true for s/z representation:

The lips MUST NOT be fully closed. Fully closed lips make s/z production impossible. (Air needs to be able to escape!)
The jaw MUST NOT be significantly open. Though it is difficult to draw the line at what might be considered “significantly open” (especially due to individual differences), this point is still necessary to keep in mind. The more the jaw opens, the more difficult it is to produce a recognizable s/z.

Vowels Overview

A vowel is a speech sound that, unlike a consonant, does not require closure, turbulence, or constriction. The vocal tract is relatively unrestricted during vowel production, and vowels are always voiced.

For our purposes, we will be focusing on these categories:

rounded vowels: lips take on a rounded shape; may be a tightly rounded shape like w/r/oo, a loosely rounded shape like ouh (ʊ), or anything in between.
relaxed vowels: lips do not take on any particular shape and remain relaxed.
wide (corner-pinched) vowels: lips extend in width (increase distance from lip corner to lip corner) usually due to a lip corner puller (smile) or dimpler.

Rules for "ouh"

The lips MUST NOT be fully closed. Fully closed lips makes "ouh" production impossible.

The lips are most natural when semi-rounded. Though the rounding for "ouh" may be subtle in everyday speech, when the lips are slightly rounded, the "ouh" sound is most recognizable.

The jaw MUST NOT be fully closed.

Made it this far? Give yourself a pat on the back and happy vubbing!