Getting your Face Capture data ready
To start with, you will need to export the facial data you have recorded in Rokoko Studio. To do this, right-click the recording you want to export and choose "Export".
In the export settings choose "FBX" as the format and make sure to check the "Face" option under "Data Included".
Under the FBX settings, change the FBX version to 2011, this is very important, as newer versions may not import correctly into Unreal.
Set any additional settings as you desire, please also note that the export skeleton does not matter for facial animations as we will later disable all skeletal data from the animation inside Unreal.
Preparing your custom character
Before we start importing things into Unreal, you will need to make sure all morph targets (also known as blend shapes or shape keys) for your characters face deformations are named correctly. These are the 51 blend shapes used for facial animation that you should already have prepared for your character before getting face animation onto them. The problem is that there often is some slight deviations in the naming of the blend shapes depending on who provided them. To fix this, please open up your custom character in an external 3D editor (Blender will work for this as free software, any other 3D editing software with blend shape features will also work) and go through the blend shapes and rename them exactly according to the naming used by Rokoko. You can find a list of the exact naming that should be used attached at the bottom of this article.
After making sure your blend shapes are named correctly, please export the character as FBX (Please make sure to also enable blend shape exports in the program you're exporting from, in most programs this is usually an option that can be toggled when exporting a model as a .fbx)
Importing your facial animation and custom character model into Unreal Engine
We will start by importing your custom character into Unreal Engine. This process is much the same as you normally would go through, with one extra important step. After making sure "Import Mesh" and "Skeletal Mesh" are enabled, open up the Mesh dropdown menu in your import settings and enable the "Import Morph Targets" option. This is how you get your custom character facial blend shapes into Unreal. The rest of the settings can be set as you wish.
After your character is imported, we will import your face capture data.
Navigate to where you chose to export your mocap data from Rokoko Studio to, there, you should find a folder with multiple .fbx files. You want to use the face data. The face data should contain the word "face" in it, so it should look something like: "yourMocapName_face_...".
Import this file into Unreal Engine.
When importing your face data you need to make sure you choose the following settings in your FBX Import Options:
- Enable "Skeletal Mesh" and "Import Mesh".
- Set the skeleton as "None".
- Open the Mesh dropdown menu and enable "Import Morph Targets".
- Enable "Import Animations".
- Open the Animation dropdown menu and disable "Import Bone Tracks".
- You can also disable all material and texture imports if you wish, as we won't be needing them.
After having imported your assets. Please make sure to save them in your Unreal project by going into "File"->"Save All" or pressing ctrl+shift+s. If you don't save your assets now it may cause issues down the line, so be sure to save everything now.
Retargetting your facial animation onto your character
You should now have the facial animation inside Unreal Engine applied to Rokoko's default Newton character mesh, but you'll likely want to have that facial animation transferred onto your own custom character, we will go over how to do so now.
Start by opening the skeleton asset that Unreal automatically generated for your face capture data when you imported it. To do this, go into the Content Browser and find the skeleton named "yourFaceDataName_skeleton" located in your import location. Double click the skeleton to open it up.
In the skeleton window, start by navigating to the "Preview Scene Settings" window and finding the "Mesh"->"Preview Mesh (Skeleton)"->"Apply To Asset" option and click it.
After doing that, while still in your skeleton window, navigate to the "Retarget Manager" window.
In it, find the "Set up Rig" tab and open the "Select Rig" dropdown menu. In the dropdown menu, select the "Select Humanoid Rig" option.
After doing this, make sure to save the changes you made to the skeleton by clicking the "Save" button in the top left corner of the skeleton window. After saving you can close down the Skeleton window.
Now we are going to retarget the facial animation onto your custom character. To do this, go into the Content Browser and find the face data animation that was created when you imported your face data into Unreal, it should be located in the same place that your Skeleton Asset was located and in the same fashion, will be named "yourFaceDataName_Anim".
Right-click the animation asset and choose "Retarget Anim Assets"->"Duplicate Anim Assets and Retarget".
This will open up a "Select Skeleton" window. In it, choose the skeleton used for your custom character asset, a preview of your character should appear in the right "[Target]" window after choosing its skeleton.
Choose the folder and any naming you want to be applied to your new retargeted animation in the right options set and leave the rest of the settings as is and click "Retarget".
You should now have a new animation asset at your set folder destination with the facial animation applied to your own character.
Playing your facial animation with skeleton animation.
The last step we will go over is how to apply your new facial animation with any skeleton animation on your character. This skeleton animation can be mocap data imported from Rokoko studio or any other animations.
To start with, we will first create a new animation blueprint.
Alternatively, if you wish, you can also open up any other animation blueprint that you want to apply facial animation to and use the same steps to apply the facial animation to it.
To create a new animation blueprint, right-click on any empty space in your Content Browser and choose "Animation"->"Animation Blueprint".
In the "Create Animation Blueprint" popup, go into the "Target Skeleton" tab and choose the skeleton of your custom character and press "OK".
Now open up the new Animation Blueprint asset by double-clicking on it.
This will open up a new window, in there, you should already be in the correct "Anim Graph" tab. In the Anim Graph tab, right-click on any empty space to create a new node, and search for the "Layered blend per bone" node and choose it.
Having created the "Layered blend per bone" node, click it and navigate into the "Details" tab. In the Details tab, expand the "Layer Setup" option by clicking the small arrow to the left of it. Expand the subsequent "0" option as well and click the "+" icon next to the "0 Array Elements" text to add a new element.
After adding an element, expand the new element with the arrow and change the "Blend Depth" option from "0" to "1". Leave the Bone Name option empty.
Now it's time to add your animations. Start by dragging your desired skeleton animation from the "Asset Browser" tab into the Anim Graph. Next, drag your desired facial animation from the Asset Browser tab into the Anim Graph.
Next, connect them both into the Layered blend per bone node. Your skeleton animation should go into the "Base Pose" node space and your face animation should go into the "Blend Poses 0" node space. Click the green "Compile" button in the animation preview window or press the "Compile" button in the top left corner of the animation blueprint window to see your combined animation playing.
Remember to save your animation blueprint when finished.
If you are using an already build blueprint which has some other animations in it already, you should be able to add the facial animation in the same manner, just plug in your animations/states into the "Base Pose" node space before plugging it into the final Output Pose to add the facial animation onto them. You can also create any conditions or state machines you want to use for your facial animation before plugging them into the Layered blend per bone node to create more advanced facial animation logic, the Layered blend per bone node is simply used to blend the facial animation onto any other bone or curve animation data you have.
You should now have a facial animation that can be applied along with any skeleton animation you want. To add extra animations, simply go through the same steps for the facial imports etc. again and add them to your character via the animation blueprint.