Face Cap App Utilities

Faceit provides an interface for the IOS Face Cap App. Find more information on the FaceCap App here. As a Face Cap user you can:

Warning

To use Live Capturing (OSC connection) for longer than 5 seconds you will have to unlock all features in the Face Cap App:
Price: 60$

Load Recorded Motion (TXT)

Face Cap motion can be recorded directly on the capturing device (e.g. the iPhone). You can either save to FBX(Ascii) or to .txt format. Simply send the saved motion from your device to your computer via Fileshare or Email or w/e.

ASCII FBX

If you want to use FBX, you can use the new FBX Retargeting tool. You got to be aware of the fact that Face Cap App exports FBX in Ascii format, while Blender does only support Binary FBX imports. There are converters and add-ons to work around this problem.

Tip

If you don't have the Face Cap app and quickly want to test the import workflow you can download a test motion here.

SetupOSC
Import motion data from .txt files directly exported by the Face Cap App. You can retarget the motion to the Control Rig within the operator.

Browse Txt file

  • Click the Load FaceCap .txt operator to browse for a .txt file.
  • Afterwards hit Import to apply it to all registered geometry.

SetupOSC
Choose a .txt file exported from Face Cap.

Options (TXT Import)

  1. Import to new Action
    • If this is enabled, Faceit will create a new Action to hold the new Keyframes.
    • Otherwise you can specify a Shape Key Action from the scene.
  2. Overwrite existing Animation
    • If you choose to import to an existing animation, Faceit will prompt to overwrite the Action. If you choose not to overwrite, Faceit will attempt to merge the Actions, overwriting only the new motions frame range (see start frame option below).
  3. Start Frame
    • Specify a Start Frame for the new animation values. Faceit will overwrite existing animation values in the new motions frame range.
    • If you specify a frame behind the current motion, the new values will be appended to the Action, preserving all current animation.
  4. Bake to Control Rig
    • If the Control Rig has been created, the operator presents the option to directly bake the new animation values onto the sliders.

Live Capturing (OSC)

Face Cap offers the possibility to stream all motion live into Blender via OSC(Open Sound Control). Lucky for us JPfeP developed the amazing AddRoutes Add-on for Blender that covers receiving and sending of OSC messages.

SetupOSC
SetupOSC
Stream Live to the Blender viewport and record an aimation.

AddRoutes Add-on

Get AddRoutes here!

SetupOSC
You have to install AddRoutes to use Live Capturing via OSC. See the link above!

Missing Modules

In Blender 2.93, the AddRoutes Add-on can't be installed, because of a missing module. You can fix this issue, by executing the following code in the Blender python consolde, before activating the add-on: 

import subprocess
import sys
subprocess.check_call([sys.executable, "-m", "pip", "install", "python-rtmidi"])

Setup OSC Routes

In order to receive the ARKit data, we need to setup the OSC routes in the AddRoutes add-on configuration. Faceit provides an operator that allows to quickly setup all routes for the registered objects. As the 52 Shape Key values (the motion) are send one by one, we have set up multi-routing. Faceit does automatically set up the routes for all registered geometry (provided that it holds the shape keys).

SetupOSC
Create the OSC Routes that allow to receive the Mocap Data in realtime.

Order Matters

The order of the ARKit shape keys matters in Face Cap Live (OSC) streaming, because the OSC routes will receive the incoming animation data (52 values) in a fixed order. Faceit will attempt to reorder the shape keys automatically to match that order, but if 1 or more target shapes are missing, the indices won't match and the wrong shapes will be animated.

  • Create all routes, by selecting the Setup FaceCap OSC Routes operator.
  • Clear all routes, by selecting the Clear OSC routes operator.

SetupOSC
This is an exemplary route for shape keys. A route like this will be created for every Shape Key object.

AddRoutes Settings

Start listening to the port:
AddR Config > OSC Config > Listen on (enable checkmark)
SetupOSC
Set up the OSC routes.

0.0.0.0 allows to listen to all the network devices of your computer, you might want to restrict that later but it's handy for tests. The connections are updated as soon as you change a parameter if enabled.

Note

Both your PC and the iPhone need to be in the same LAN (local area network). You might also have to allow access through your Firewall!

Put Face Cap into Live Mode

Before you can receive anything in Blender you have to put the Face Cap app into Live Mode.

  • Enter the IP-address of the computer that runs your Blender instance.
  • Enter the same port as in AddRoutes Settings

SetupOSC
Enter IP address of your PC and the same port as in AddRoutes.

Record

Warning

Depending on the complexity of your scene and your network connection, AddRoutes might not be able to record the motion in realtime. See Improve Live Performance

Clicking the Record button will enable recording of the motion. To record follow these steps:

  1. Press Record
  2. Put the FaceCap into Live mode
  3. Make sure the connection is working by holding the iPhone in front of your face.
  4. Hit Play in Blender and all motions will be keyframed.

Improve Live Performance

There are multiple factors that contribute to the quality and speed of realtime animation in the viewport! See the tips box below!

Useful tips for better Scene Performance

  1. Complex Scene
  2. Rendermode
    • Eevee shaders, and even more so Cycles viewport render, can slow down playback immensely! Check if it works faster in solid render mode.
  3. Custom Normals and Auto Smooth:
  4. Hide Objects:
    • Hide complex geometry from the viewport. This improves performance, because otherwise blender will update the object data on every frame.
  5. Checkout the other tips on performance in the ARP documentation:
  6. The OSC data is streamed over your Wifi network. You should aim to optimize streaming performance. Usually this shouldn't be the bottleneck on modern setups.

simpl
Try to enable the Simplify Setting and lower the number of max subdivisions to 0.

Order & Indices

#      FaceCap          ARKit     
    [00]         browInnerUp          eyeBlinkLeft     
    [01]         browDownLeft          eyeLookDownLeft     
    [02]         browDownRight          eyeLookInLeft     
    [03]         browOuterUpLeft          eyeLookOutLeft     
    [04]         browOuterUpRigh          eyeLookUpLeft     
    [05]         eyeLookUpLeft          eyeSquintLeft     
    [06]         eyeLookUpRight          eyeWideLeft     
    [07]         eyeLookDownLeft          eyeBlinkRight     
    [08]         eyeLookDownRigh          eyeLookDownRigh     
    [09]         eyeLookInLeft          eyeLookInRight     
    [10]         eyeLookInRight          eyeLookOutRight     
    [11]         eyeLookOutLeft          eyeLookUpRight     
    [12]         eyeLookOutRight          eyeSquintRight     
    [13]         eyeBlinkLeft          eyeWideRight     
    [14]         eyeBlinkRight          jawForward     
    [15]         eyeSquintLeft          jawLeft     
    [16]         eyeSquintRight          jawRight     
    [17]         eyeWideLeft          jawOpen     
    [18]         eyeWideRight          mouthClose     
    [19]         cheekPuff          mouthFunnel     
    [20]         cheekSquintLeft          mouthPucker     
    [21]         cheekSquintRigh          mouthRight     
    [22]         noseSneerLeft          mouthLeft     
    [23]         noseSneerRight          mouthSmileLeft     
    [24]         jawOpen          mouthSmileRight     
    [25]         jawForward          mouthFrownRight     
    [26]         jawLeft          mouthFrownLeft     
    [27]         jawRight          mouthDimpleLeft     
    [28]         mouthFunnel          mouthDimpleRigh     
    [29]         mouthPucker          mouthStretchLef     
    [30]         mouthLeft          mouthStretchRig     
    [31]         mouthRight          mouthRollLower     
    [32]         mouthRollUpper          mouthRollUpper     
    [33]         mouthRollLower          mouthShrugLower     
    [34]         mouthShrugUpper          mouthShrugUpper     
    [35]         mouthShrugLower          mouthPressLeft     
    [36]         mouthClose          mouthPressRight     
    [37]         mouthSmileLeft          mouthLowerDownL     
    [38]         mouthSmileRight          mouthLowerDownR     
    [39]         mouthFrownLeft          mouthUpperUpLef     
    [40]         mouthFrownRight          mouthUpperUpRig     
    [41]         mouthDimpleLeft          browDownLeft     
    [42]         mouthDimpleRigh          browDownRight     
    [43]         mouthUpperUpLef          browInnerUp     
    [44]         mouthUpperUpRig          browOuterUpLeft     
    [45]         mouthLowerDownL          browOuterUpRigh     
    [46]         mouthLowerDownR          cheekPuff     
    [47]         mouthPressLeft          cheekSquintLeft     
    [48]         mouthPressRight          cheekSquintRigh     
    [49]         mouthStretchLef          noseSneerLeft     
    [50]         mouthStretchRig          noseSneerRight     
    [51]         tongueOut          tongueOut