Refinement Video Series
Understanding the Interface
Buttons and Controls
Blendshapes
Applying Blendshapes
Best Practices
Advanced Settings
This article focuses on the technical aspects of the refinement tool and how to fix mouth movements.
Refinement Video Series
Please watch our 3-part series on how to use the Refinement tool in DeepEditor.
DeepEditor Refinement: Chapter 1
Deep Editor Refinement: Chapter 2
DeepEditor Refinement: Chapter 3
Read "Lipsync Theory" to learn to spot issues with your vubs' mouth movements.
Understanding the Interface
Main Viewer
The main viewer is where your Vubbed shot is displayed.
Properties Panel
The properties panel is where you can edit driving data and blendshapes.
Reference Panel
The reference panel is where you can see the following elements:
- PT Result: This is a gray face that allows you to preview the refinement changes you are making to the character without having to render the shot.
- Driving data: This gray face allows you to see how the AI is interpreting the driving data.
- Plate Proxy: This image displays the original shot uploaded into the project. It is the same as “Source Media”
Timeline
The timeline is where you get a visual representation of how you are altering your vub. With every move on the Properties Panel, you will see your timeline change.
On the timeline, you can see:
- Audio waveform
- Keyframes
- Blendshapes in use
Buttons and Controls
List Menu
With the list menu at the top right corner you can do multiple actions:
- Navigate to Recent Projects
- Open a tab to Frequently Asked Questions
- Get information about how to Contact Support
- Log out of your user account
Playback
Play controls are located at the top left of the timeline
Click the icon to play the entirety of your vub.
3D Mask
When you open a shot you will see a 3D gray mask on your vubbed character.
This mask allows you to see the PT refinement you are making without rendering.
Click the icon to have the toggle the gray mask on and off.
Keyframes
What is a keyframe?
Keyframes are markers that indicate the beginning and end of an action in an animation, effect, or motion graphic.
How do you adjust a keyframe?
Slider Adjustment: Click and drag the slider to adjust keyframes.
Numerical Adjustment: Type in a number value, positive or negative to adjust a keyframe.
Any adjustment made with these tools will affect the frame your playhead is resting on. If there is no current keyframe, one will be created.
Interpolation
Ranging from 0 to 100, this represents the effect that the Driving Data mouth movement has on the original mouth movement: ‘0' means that there is zero application of the Driving Data (original performance is completely retained), while '100’ would mean that the Driving Data mouth movement entirely replaces the original movements.
Keep in mind, any changes made with 0% Interpolation will not appear in the Output Media.
Window size adjustment
If you wish to adjust the windows in the DeepEditor refinement tool, click and drag these icons to your desired location.
Slapcomp Media vs Source Media views
When opening the refinement page you will automatically see the “Output Media”. If you want to see the original mouth movements of your character you can toggle to view the “Source Media” by clicking this drop down.
Zoom on View Windows
If you are having trouble viewing a character's mouth, especially if they are far from the camera, you can zoom in.
- Mouse click = Zoom 150%
- Click again to revert to 100%
- Mouse hover and move scroll wheel = Zoom in increments.
- Mouse hover, click and hold = Allows you to adjust the picture in PT Monitor.
After zooming in, a new button will appear at the bottom center of the viewers, “Reset Zoom” click to reset the position and zoom back to standard.
Undo/Redo
Click this icon to undo your latest action. You may click multiple times to go back further.
Click this icon to redo an undone action.
Render changes
When you want to see how your refinement changes are affecting your character's real face, not just the 3D mask, hit “Render changes”.
The time it takes to render changes can differ based on the length and complexity of your vub. At present, rendering an average vub typically takes about 10 minutes.
Save and Load
Once you are satisfied with your work, click the "Save" button. You will be asked to name your save. It's advisable to include a version number in your save name. For instance: Horizon_Vub_V02.
To revert to a previous version, click “Load” and you will be asked to select an earlier version.
Download latest render
Once you are happy with your refinement and want to see the changes in your NLE timeline, hit “Download latest render”.
Download times may vary based on the length and complexity of your vub. The average download will take less than a minute.
The vub file will appear in your computer system’s "Downloads" folder.
Blendshapes
Blendshapes are the tools in DeepEditor that allow you to finetune the mouth movements of an actor, so you can refine a vub to your liking.
DeepEditor focuses on the lower half of the face (nose to chin).
The more of the original performance is preserved, the less noticeable your overrides will be.
Each blendshape can be given a negative (-) value. This will affect the face of the opposing movement. For example, setting Jaw Open to -10 will close the Jaw. Setting the funnel to -10 will make the mouth less round.
Applying Blendshapes
Move the playhead to the frame you’d like to adjust.
To the left of the timeline you will find your active tools.
Your keyframe adjustments are displayed inside of the timeline.
You can add multiple Blendshapes to the same frames.
Beware! Too many can cause an overworked and unnatural looking Vub
Note: You will not be able to see your adjustments on the model until you click “Render Changes.” The Grayface under PT Result will show live results.
They operate as a typical keyframe, adding an effect to a range of frames between its in and out points.
Your active Blendshapes will populate to the left of the timeline.
Best Practices
Stick to the Source Media as much as possible.
If the vub is slightly out of sync, an NLE audio edit or nudge is a good starting point.
For example: If the character only speaks at the beginning of the shot, ease the Interpolation from 100 to 0 after the line.
Use a “light touch”
The intention is that there shouldn’t be a need to add a large number of keyframes to the timeline: the majority of the final PT Result should be derived automatically by DeepEditor
Heavy movements can look unnatural. Start with 10 or -10. Blend shape adjustments should rarely go above 30 or -30.
Don’t overwork the shot. Too many refinements can make the output media look worse, not better.
Smoothly Transition
One or two frames is not enough time for a mouth to completely change. “Ride the wave” and try to take at least 4 frames to transition.
Note: This is not a “rule”. Sometimes a character may speak very quickly and need sudden changes.
Light moves faster than sound.
Starting a movement a frame or two early may look better to the viewer.
Advanced Settings
Window Names (Advanced)
Navigation bar
Here you can navigate back to the Home or Project pages.
User Display
Here is a representation of who is logged in currently.
Frame Rate Display
At the top right corner of the timeline the frame rate is displayed. This is the same frame rate as your source media. You can not change the frame rate of your vub in DeepEditor.
Timecode Display
Above you can see at what timecode your playhead is rested. Note: this is not the timecode of your source media but the vubbed shot itself.
Buttons and Controls (Advanced)
“Go to frame” function
Click on the frame count display to open a pop-up menu. Enter a specific frame number, and your playhead will move to that frame.
Change Reference Panel Views
In the reference panel there are 3 views. You can mix and match your views to your preference using this dropdown menu.
Change playback speed
In the top left corner of your timeline, you'll find the word "Normal," indicating that your timeline is playing at regular speed. You can alter this by clicking to reveal a dropdown menu.
Turn off loop video playback
If you'd wish your playback to stop looping, click this icon at the top left corner of the timeline. The vub will stop at the end.
Mute
Want to mute the timeline? Click the speaker icon at the top left corner of the timeline. You can turn the audio back on by clicking the button again.
Frame forward/backward buttons
Click these buttons to step one frame forward and one frame backward in the timeline.
Lock Blendshapes in the Timeline
The lock icon below allows you to secure a specific blend shape or driving data element in the timeline. This is useful for making minor adjustments to one blend shape without inadvertently altering another.
Hide Blendshapes in the Timeline
Above you will see a circle icon. These allow the user to hide a blend shape or driving data element in the timeline.
This is particularly helpful if you want to concentrate on a single keyframe path. Rest assured, hidden keyframes cannot be modified.
Timeline Zoom
You can zoom in on a specific frame range on the timeline by:
- Clicking and dragging the adjustment bar at the top.
- Typing the desired number in the boxes at the ends.
- Revert to the default display by clicking the arrow key.
Keyframe Buttons
When your playhead hovers over a keyframe in the timeline, these diamond icons become filled. They not only show the location of a keyframe but can also be used to create one.
Click the icon to generate a keyframe with a value of 0.
Show Advanced Blendshapes
Clicking the "Show advanced blendshapes" toggle button will grant you access to all of the following:
Keyboard shortcuts
General Shortcuts
Ctrl + Z = Undo previous action
Ctrl + Y = Redo previous action
Timeline Shortcuts
Spacebar = Play/Pause
⬅️ ➡️ Right/Left Arrows = Move playhead right/left one frame
Shift + Right/Left Arrow = Move the playhead to keyframe to the right/left
Option + Right/Left Arrow = Move the playhead to the right/left 10 frames
⬆️ Up Arrow = Jump the playhead to the first frame
⬇️Down Arrow = Jump the playhead to the last frame
Click + Drag Mouse = Bounding box to select multiple keyframes
Selected Keyframe + Backspace = Delete a keyframe.
⬅️ ➡️ Selected Keyframe + Right/Left Arrow = Moves keyframe to the right/left one frame.
Selected Keyframe + Up/Down Arrow = Moves keyframe up/down one number value.
Selected Keyframe + Shift + Up/Down Arrow = Moves keyframe up/down 10 number values.
Selected Keyframe + Shift + Right/Left Arrow = Moves keyframe right/left 10 frames
PT Main Viewer & Reference Panel Viewers
Mouse click = Zoom 50%
Mouse hover and move scroll wheel = Zoom in increments.
Mouse hover, click and hold = Allows you to adjust the picture in PT Monitor.