Face Cap App Utilities
Faceit provides an interface for the IOS Face Cap App. Find more information on the FaceCap App here. As a Face Cap user you can:
- Load motion (.txt format) from the Face Cap App. Read more on loading Recorded Motions here!
- Stream the captured motion live into Blender. Read more on the Live Capturing feature here!
Warning
To use Live Capturing (OSC connection) for longer than 5 seconds you will have to unlock all features in the Face Cap App:
Price: 60$
Motion Types
Next to the shape key values, the app also records rotation values for the head and eyes. To load these values along with the shape key animation, you will need to activate the motion types in the general settings
Load Recorded Motion (TXT)
Face Cap motion can be recorded directly on the capturing device (e.g. the iPhone). You can either save to FBX(Ascii) or to .txt format. Simply send the saved motion from your device to your computer via Fileshare or Email or w/e.
ASCII FBX
If you want to use FBX, you can use the new FBX Retargeting tool. You got to be aware of the fact that Face Cap App exports FBX in Ascii format, while Blender does only support Binary FBX imports. There are converters and add-ons to work around this problem.
Tip
If you don't have the Face Cap app and quickly want to test the import workflow you can download a test motion here.
Import motion data from .txt files directly exported by the Face Cap App. You can retarget the motion to the Control Rig within the operator.
Browse Txt file
- Click the
Load FaceCap .txt
operator to browse for a .txt file. - Afterwards hit Import to apply it to all registered geometry.
Choose a .txt file exported from Face Cap.
Options (TXT Import)
- Import to new Action
- If this is enabled, Faceit will create a new Action to hold the new Keyframes.
- Otherwise you can specify a Shape Key Action from the scene.
- Overwrite existing Animation
- If you choose to import to an existing animation, Faceit will prompt to overwrite the Action. If you choose not to overwrite, Faceit will attempt to merge the Actions, overwriting only the new motions frame range (see start frame option below).
- Start Frame
- Specify a Start Frame for the new animation values. Faceit will overwrite existing animation values in the new motions frame range.
- If you specify a frame behind the current motion, the new values will be appended to the Action, preserving all current animation.
- Bake to Control Rig
- If the Control Rig has been created, the operator presents the option to directly bake the new animation values onto the sliders.
Live Capturing (OSC)
Face Cap offers the possibility to stream all motion live into Blender via OSC(Open Sound Control). Lucky for us JPfeP developed the amazing AddRoutes Add-on for Blender that covers receiving and sending of OSC messages.
Stream Live to the Blender viewport and record an aimation.
AddRoutes Add-on
You have to install AddRoutes to use Live Capturing via OSC. See the link above!
Missing Modules
In Blender 2.93, the AddRoutes Add-on can't be installed, because of a missing module. You can fix this issue, by executing the following code in the Blender python consolde, before activating the add-on:
import subprocess
import sys
subprocess.check_call([sys.executable, "-m", "pip", "install", "python-rtmidi"])
Setup OSC Routes
In order to receive the ARKit data, we need to setup the OSC routes in the AddRoutes add-on configuration. Faceit provides an operator that allows to quickly setup all routes for the registered objects. As the 52 Shape Key values (the motion) are send one by one, we have set up multi-routing. Faceit does automatically set up the routes for all registered geometry (provided that it holds the shape keys).
Create the OSC Routes that allow to receive the Mocap Data in realtime.
Order Matters
The order of the ARKit shape keys matters in Face Cap Live (OSC) streaming, because the OSC routes will receive the incoming animation data (52 values) in a fixed order. Faceit will attempt to reorder the shape keys automatically to match that order, but if 1 or more target shapes are missing, the indices won't match and the wrong shapes will be animated.
- Create all routes, by selecting the
Setup FaceCap OSC Routes
operator. - Clear all routes, by selecting the
Clear OSC routes
operator.
This is an exemplary route for shape keys. A route like this will be created for every Shape Key object.
AddRoutes Settings
Start listening to the port:
AddR Config > OSC Config > Listen on (enable checkmark)
Set up the OSC routes.
0.0.0.0 allows to listen to all the network devices of your computer, you might want to restrict that later but it's handy for tests. The connections are updated as soon as you change a parameter if enabled.
Note
Both your PC and the iPhone need to be in the same LAN (local area network). You might also have to allow access through your Firewall!
Put Face Cap into Live Mode
Before you can receive anything in Blender you have to put the Face Cap app into Live Mode.
- Enter the IP-address of the computer that runs your Blender instance.
- Enter the same port as in AddRoutes Settings
Enter IP address of your PC and the same port as in AddRoutes.
Record
Warning
Depending on the complexity of your scene and your network connection, AddRoutes might not be able to record the motion in realtime. See Improve Live Performance
Clicking the Record button will enable recording of the motion. To record follow these steps:
- Press Record
- Put the FaceCap into Live mode
- Make sure the connection is working by holding the iPhone in front of your face.
- Hit Play in Blender and all motions will be keyframed.
Improve Live Performance
There are multiple factors that contribute to the quality and speed of realtime animation in the viewport! See the tips box below!
Useful tips for better Scene Performance
- Complex Scene
- Maybe you have a subdivision surface modifier in the stack or a multires modifier. I like to use the simplify settings to quickly toggle all modifiers on/off. Click here to learn more about simplify settings
- Rendermode
- Eevee shaders, and even more so Cycles viewport render, can slow down playback immensely! Check if it works faster in solid render mode.
- Custom Normals and Auto Smooth:
- These functions are very demanding and can slow playback performance immensely!
- This neat little script by Artell will quickly remove them from your scene.
- Hide Objects:
- Hide complex geometry from the viewport. This improves performance, because otherwise blender will update the object data on every frame.
- Checkout the other tips on performance in the ARP documentation:
- The OSC data is streamed over your Wifi network. You should aim to optimize streaming performance. Usually this shouldn't be the bottleneck on modern setups.
Try to enable the Simplify Setting and lower the number of max subdivisions to 0.
Order & Indices
# | FaceCap | ARKit |
---|---|---|
[00] | browInnerUp | eyeBlinkLeft |
[01] | browDownLeft | eyeLookDownLeft |
[02] | browDownRight | eyeLookInLeft |
[03] | browOuterUpLeft | eyeLookOutLeft |
[04] | browOuterUpRigh | eyeLookUpLeft |
[05] | eyeLookUpLeft | eyeSquintLeft |
[06] | eyeLookUpRight | eyeWideLeft |
[07] | eyeLookDownLeft | eyeBlinkRight |
[08] | eyeLookDownRigh | eyeLookDownRigh |
[09] | eyeLookInLeft | eyeLookInRight |
[10] | eyeLookInRight | eyeLookOutRight |
[11] | eyeLookOutLeft | eyeLookUpRight |
[12] | eyeLookOutRight | eyeSquintRight |
[13] | eyeBlinkLeft | eyeWideRight |
[14] | eyeBlinkRight | jawForward |
[15] | eyeSquintLeft | jawLeft |
[16] | eyeSquintRight | jawRight |
[17] | eyeWideLeft | jawOpen |
[18] | eyeWideRight | mouthClose |
[19] | cheekPuff | mouthFunnel |
[20] | cheekSquintLeft | mouthPucker |
[21] | cheekSquintRigh | mouthRight |
[22] | noseSneerLeft | mouthLeft |
[23] | noseSneerRight | mouthSmileLeft |
[24] | jawOpen | mouthSmileRight |
[25] | jawForward | mouthFrownRight |
[26] | jawLeft | mouthFrownLeft |
[27] | jawRight | mouthDimpleLeft |
[28] | mouthFunnel | mouthDimpleRigh |
[29] | mouthPucker | mouthStretchLef |
[30] | mouthLeft | mouthStretchRig |
[31] | mouthRight | mouthRollLower |
[32] | mouthRollUpper | mouthRollUpper |
[33] | mouthRollLower | mouthShrugLower |
[34] | mouthShrugUpper | mouthShrugUpper |
[35] | mouthShrugLower | mouthPressLeft |
[36] | mouthClose | mouthPressRight |
[37] | mouthSmileLeft | mouthLowerDownL |
[38] | mouthSmileRight | mouthLowerDownR |
[39] | mouthFrownLeft | mouthUpperUpLef |
[40] | mouthFrownRight | mouthUpperUpRig |
[41] | mouthDimpleLeft | browDownLeft |
[42] | mouthDimpleRigh | browDownRight |
[43] | mouthUpperUpLef | browInnerUp |
[44] | mouthUpperUpRig | browOuterUpLeft |
[45] | mouthLowerDownL | browOuterUpRigh |
[46] | mouthLowerDownR | cheekPuff |
[47] | mouthPressLeft | cheekSquintLeft |
[48] | mouthPressRight | cheekSquintRigh |
[49] | mouthStretchLef | noseSneerLeft |
[50] | mouthStretchRig | noseSneerRight |
[51] | tongueOut | tongueOut |