Refine a Vub

This article focuses on the technical aspects of the refinement tool and how to fix mouth movements.

Here are a few videos to help your understanding.

DeepEditor Refinement: Chapter 1

Deep Editor Refinement: Chapter 2

DeepEditor Refinement: Chapter 3

Read "Lipsync Theory" if you'd like to learn to spot issues with your vubs mouth movements.

🔽 Download Your Vub

Once your asset has finished generating, the status will change to "Completed". You will gain the option to "Download" and/or "Refine" your asset.

You can click "Review and Download" from the automated email once your DeepEditor vub is done.

Locate your assets in the project

Beginner Information:

Window Names (Beginner)

🖼️ Main Viewer

The main viewer is where your Vubbed shot is displayed.

Properties Panel

The properties panel is where you can edit driving data and blend shapes.

Reference Panel

The reference panel is where you can see the following elements:

PT Result: This is a gray face that allows you to preview the refinement changes you are making to the character without having to render the shot.
Driving data: This gray face allows you to see how the AI is interpreting the driving data.
Plate Proxy: This image displays the original shot uploaded into the project. It is the same as “Source Media”

Timeline

The timeline is where you get a visual representation of how you are altering your vub. With every move on the Properties Panel, you will see your timeline change.

On the timeline, you can see:

Audio waveform
Keyframes
Blendshapes in use

Buttons and Controls (Beginner)

📋 List Menu

With the list menu at the top right corner you can do multiple actions:

Navigate to Recent Projects
Open a tab to Frequently Asked Questions
Get information about how to Contact Support
Log out of your user account

▶️ Playback

Play controls are located at the top left of the timeline

Click the icon to play the entirety of your vub.

3D Mask

When opening a shot there will be a 3D gray mask on your vubbed character.

This mask allows you to see the PT refinement you are making without rendering.

Click the icon to have the gray mask in view or not.

🔷 Keyframes

What is a keyframe?

Markers that indicate the beginning and end of an action in an animation, effect, or motion graphic.

How do you adjust a keyframe?

Slider Adjustment: Click and drag the slider to adjust keyframes.

Numerical Adjustment: Type in a number value, positive or negative to adjust a keyframe.

Any adjustment made with these tools will affect the frame you playhead is resting on. If there is no current keyframe, one will be created.

📉 Interpolation

Ranging from 0 to 100, this represents the effect that the Driving Data mouth movement has on the original mouth movement: ‘0' means that there is zero application of the Driving Data (original performance is completely retained), while '100’ would mean that the Driving Data mouth movement entirely replaces the original movements.

Keep in mind, any changes made with 0% Interpolation will not appear in the Output Media.

🖼️ Window size adjustment

If you wish to adjust the windows in the DeepEditor refinement tool, click and drag these icons to your desired location.

👀 Slapcomp Media vs Source Media views

When opening the refinement page you will automatically see the “Output Media”. If you want to see the original mouth movements of your character you can toggle to view the “Source Media” by clicking this drop down.

🔍 Zoom on View Windows

If you are having trouble viewing a character's lower mouth, especially if they are far from the camera, you can zoom in.

Mouse click = Zoom 150%
Click again to revert to 100%
Mouse hover and move scroll wheel = Zoom in increments.
Mouse hover, click and hold = Allows you to adjust the picture in PT Monitor.

After zooming in, a new button will appear at the bottom center of the viewers, “Reset Zoom” click to reset the position and zoom back to standard.

↩️ Undo/Redo

Click this icon to undo your latest action. You may click multiple times to go back further.

Click this icon to redo an undone action.

⚡ Render changes

When you would like to see how your refinement changes are affecting your character's real face, not just the 3D mask, hit “Render changes”.

Depending on the length and complexity of your vub, render changes may vary. Currently an average vub will take 10 minutes to render.

💾 Save and Load

When you are happy with your work, hit the “Save” button.

You will be prompted to name your save.

We recommend that you name your saves with some kind of version number.
Example: Horizon_Vub_V02

If you wish to go back to your saved version, click “Load” and you will be prompted to choose old work.

⏬ Download latest render

Once you are happy with your refinement and want to see the changes in your NLE timeline, hit “Download latest render”.

Depending on the length and complexity of your vub, download times may vary. The average download will take less than a minute.

The vub file will appear in your computer system’s downloads folder.

Blendshapes (Beginner)

Blend shapes are the tools in DeepEditor that enable a user to further adjust the mouth movements of an actor. With these you can refine a vub to your liking.

Deep Editor focuses on the lower half of the face (Nose to Chin).

Blend shape overrides will be added to the scaled driving data using this method.

The more of the original performance preserved, the less noticeable your overrides are going to be.

Each Blend Shape can be given a negative (-) value. This will affect the face of the opposing movement. For example setting Jaw Open at -10 will close the Jaw. Setting the funnel at -10 will make the mouth less round.

Applying Blendshapes

Move the playhead to the frame you’d like to adjust.

To the left of the timeline you will find your active tools.

Your keyframe adjustments are displayed inside of the timeline.

You can add multiple Blendshapes to the same frames.

Beware! Too many can cause an overworked and unnatural looking Vub

Note: You will not be able to see your adjustments on the model until you click “Render Changes.” The Grayface under PT Result will show live results.

They operate as a typical keyframe, adding an effect to a range of frames between its in and out points.

Your active Blendshapes will populate to the left of the timeline.

🏆 Best Practices

Stick to the Source Media as much as possible.

If the vub is slightly out of sync, an NLE audio edit or nudge is a good starting point.

For example: If the character only speaks at the beginning of the shot, ease the Interpolation from 100 to 0 after the line.

👇 Use a “light touch”

The intention is that there shouldn’t be a need to add a large number of keyframes to the timeline: the majority of the final PT Result should be derived automatically by DeepEditor

Heavy movements can look unnatural. Start with 10 or -10. Blend shape adjustments should rarely go above 30 or -30.

Don’t overwork the shot. Too many refinements can make the output media look worse, not better.

🌊 Smoothly Transition

One or two frames is not enough time for a mouth to completely change. “Ride the wave” and try to take at least 4 frames to transition.

Note: This is not a “rule”. Sometimes a character may speak very quickly and need sudden changes.

💡Light moves faster than sound.

Starting a movement a frame or two early may look better to the viewer.

Advanced Information:

Window Names (Advanced)

🌐 Navigation bar

Here you can navigate back to the Home or Project pages.

🚹 User Display

Here is a representation of who is logged in currently.

🎞️ Frame Rate Display

At the top right corner of the timeline the frame rate is displayed. This is the same frame rate as your source media. You can not change the frame rate of your vub in DeepEditor.

🕓 Timecode Display

Above you can see at what timecode your playhead is rested. Note: this is not the timecode of your source media but the vubbed shot itself.

Buttons and Controls (Advanced)

🖼️ “Go to frame” function

Click on the frame count display to open a pop-up menu. Enter a specific frame number, and your playhead will move to that frame.

👀 Change Reference Panel Views

In the reference panel there are 3 views. You can mix and match your views to your preference using this dropdown menu.

⏩ Change playback speed

In the top left corner of your timeline, you'll find the word "Normal," indicating that your timeline is playing at regular speed. You can alter this by clicking to reveal a dropdown menu.

change speed big pic-1

🔁 Turn off loop video playback

If you'd wish your playback to stop looping, click this icon at the top left corner of the timeline. The vub will stop at the end.

🔕 Mute

Want to mute the timeline? Click the speaker icon at the top left corner of the timeline. You can turn the audio back on by clicking the button again.

▶️ Frame forward/backward buttons

Click these buttons to step one frame forward and one frame backward in the timeline.

🔒 Lock Blend Shapes in Timeline

The lock icon below allows you to secure a specific blend shape or driving data element in the timeline. This is useful for making minor adjustments to one blend shape without inadvertently altering another.

🙈 Hide Blend Shapes in Timeline

Above you will see a circle icon. These allow the user to hide a blend shape or driving data element in the timeline.

This is particularly helpful if you want to concentrate on a single keyframe path. Rest assured, hidden keyframes cannot be modified.

🔍 Timeline Zoom

You can zoom in on a specific frame range on the timeline by:

Clicking and dragging the adjustment bar at the top.
Typing the desired number in the boxes at the ends.
Revert to the default display by clicking the arrow key.

🔷 Keyframe Buttons

When your playhead hovers over a keyframe in the timeline, these diamond icons become filled. They not only show the location of a keyframe but can also be used to create one.

Click the icon to generate a keyframe with a value of 0.

➕ Show Advanced Blend Shapes

Clicking the "Show advanced blend shapes" toggle button will grant you access to all of the following:

⌨️ Keyboard short cuts

GENERAL:

Ctrl + Z = Undo previous action

Ctrl + Y = Redo previous action

TIMELINE:

Spacebar = Play/Pause

⬅️ ➡️ Right/Left Arrows = Move playhead right/left one frame

Shift + Right/Left Arrow = Move the playhead to keyframe to the right/left

Option + Right/Left Arrow = Move the playhead to the right/left 10 frames

⬆️ Up Arrow = Jump the playhead to the first frame

⬇️Down Arrow = Jump the playhead to the last frame

Click + Drag Mouse = Bounding box to select multiple keyframes

Selected Keyframe + Backspace = Delete a keyframe.

⬅️ ➡️ Selected Keyframe + Right/Left Arrow = Moves keyframe to the right/left one frame.

Selected Keyframe + Up/Down Arrow = Moves keyframe up/down one number value.

Selected Keyframe + Shift + Up/Down Arrow = Moves keyframe up/down 10 number values.

Selected Keyframe + Shift + Right/Left Arrow = Moves keyframe right/left 10 frames

🔍 PT MAIN VIEWER & REFERENCE PANEL VIEWERS:

Mouse click = Zoom 50%

Mouse hover and move scroll wheel = Zoom in increments.

Mouse hover, click and hold = Allows you to adjust the picture in PT Monitor.