What is ZivaRT?

ZivaRT (Ziva Real Time) is a machine learning based technology that allows a user to get nearly film-quality shape deformation results in real time. In particular, the software takes in a set of representative high-quality mesh shapes and poses, and trains a machine-learning model to learn how to deform and skin the mesh. ZivaRT allows for near film quality characters to be deployed in applications in real time.

Where do I get high-quality shapes for my character?

Designing a high quality character can be done in a variety of methods.

  1. Using physics-based character creation tools, such as ZivaVFX.

  2. Constructing character shapes by hand using modeling software, such as Maya, Blender, Houdini, etc.

  3. Use photogrammetry to get scans of your character, the convert those scans to time-corresponding meshes.

The shape data you provide is your character’s “ground truth” which the ZivaRT will try to match. The higher-quality your input data, the higher-quality results ZivaRT will produce, but in real time!

Note, because ZivaRT relies on a machine learning based approach, the input data required to train the ZivaRT character model must be a representative set of the motions the character will be posed in. For example, having a ZivaRT character climb a ladder when they were only trained with shapes of the character walking will not produce the desired look. Be sure to have at least some training shapes of the character in each “type” of pose to make it representative, such as walking, running, climbing, jumping, etc.

Why use ZivaRT (Ziva Real Time)?

Film quality character deformations are challenging to visualize in real time.

In the visual effects industry, time can be used to perfect the shapes of characters for a trailer, film, or any other non real-time application. Physics-based characters work well for producing high quality characters in this space. With many major films using ZivaVFX, the VFX industry has adopted the physics-based simulation of characters as the standard for film quality virtual characters.

For real-time applications, most characters use skeletal animation. The most popular algorithm for real time character skinning is Linear Blend Skinning (LBS). LBS does not include a notion of the physical properties of a character’s body as it moves through the world, causing noticeable artifacts.

There are a variety of solutions crafted to try to correct the artifacts present in linear blend skinning. For example, corrective blend shapes (also called “morph targets”) is an approach that allows one to reshape a mesh based on a parameter. This approach is inherently a manual process, requiring at least one blend shape for each problem that needs to be corrected for the entire range of motion for the character. This becomes a time-consuming and error-prone process.

ZivaRT handles these challenges with a powerful simplicity. Give the ZivaRT trainer application the shapes you want your character to have in a representative set of poses (not the entire range of motion), and your new ZivaRT character will have high-quality shapes across the range of motion with no manual effort.

High-Level Workflow

The high-level workflow is as follows:

  1. Import a collection representative training examples, which are pairs of skeleton animations (.fbx), and example mesh animations (.abc).

  2. The ZivaRTT (Ziva Real Time Trainer) trains the machine-learning model.

  3. The trained model can be visually compared against the example animations across any set of new poses, to inspect the quality of the result. The trained model can be re-trained with changed parameters or inputs as many times as you would like.

  4. A trained model can be saved to a binary file (.zrt).

  5. The character (.fbx) and the trained ZivaRT model file (.zrt) can be loaded into Maya or Unreal Engine to visualize the trained asset’s deformations in real time, under new motions not seen during training.