Tracking issues are instances of DeepEditor’s face detection and tracking missing frames where the character’s face appears. When this happens on frames where the character’s performance needs to be altered, it can cause issues with the vub because DeepEditor is then unable to make any changes to those frames.
Undersanding Tracking Issues
DeepEditor relies on advanced face detection and tracking to apply visual dubbing (vubs) with accuracy and consistency. While these systems are robust, there are natural limitations to what can be achieved given the complexity of film footage.
Occasionally, a face may not be detected in certain frames. This can happen for several reasons, such as low lighting, partial occlusion (when the face is blocked by another object or person), or very rapid movement that causes the face to blur. In these instances, DeepEditor may be unable to create a track, which means no vub adjustments can be applied to those specific frames.
Similarly, when a track is established, it may not always be perfectly smooth. Situations such as fast head movements, high-profile (side-on) angles, or other challenging camera perspectives can introduce small shakes in the track. These tracking variances can sometimes result in rendering discrepancies in certain frames.
These cases are part of the current limitations of face detection and tracking technology. Our team is continually improving these systems, but it’s important to set expectations: DeepEditor performs best when faces are well-lit, unobstructed, and captured in stable, forward-facing shots.
The first step to dealing with tracking issues is to identify them before generating a vub. This is not always possible, so below we provide you with strategies for dealing with these issues before and after generating a vub.
Identifying Tracking Issues
Once you have uploaded your media and its status has changed from Detecting characters to Ready for vub, go to the Characters tab to review:
- all the characters detected in the shot
- the number of frames they are tracked in
- the frame range within which they are tracked
If DeepEditor has missed frames where the character appears, you will know before running a vub that there will be tracking issues.
In the example below, you can see that we are missing frames 0041 and 0042, so we know the face has not been tracked on those frames, and therefore they cannot be vubbed, unless we fix the tracking issue.
Fixing Tracking Issues Before Generating a Vub
Once you have established that your shot has tracking issues, try to fix them before generating a vub, so you can avoid using up a token.
Possible Solutions
Cropping & TrimmingOne possible solution is to crop your shot, so it only includes the face you need to vub and trimming off frames where the character does not appear, then uploading this to DeepEditor. Note that this needs to be performed outside of DeepEditor.
BrighteningIf the original shot is too dark, brightening the shot using a color transform can often fix the tracking problems. Complete this outside of DeepEditor and then re-upload your shot. Then, once you have generated your vub, exported your shot and brought it back into your edit, make sure to reverse the color transform.
Re-uploading the mediaIn some cases, re-uploading the media can fix the tracking issues. This can deal with cases where a temporary bug or media upload corruption caused the track to fail.
Fixing Tracking Issues After Generating a Vub
If tracking issues appear after you have generated a vub, you can use the Refine tool to mitigate these or export a VFX turnover package, so these can be fixed by a VFX artist.