Building the 3D Asset-to-Tensor Foundation for Motion Models

Built and maintained MCP, a shared package that converts heterogeneous FBX and GLB motion assets into validated, training-ready tensor representations.

Overview

When I joined Cinamon, the AI Lab had only recently been established, and there was no existing tool that could convert 3D motion data into tensors suitable for model training. Before model development could begin, we first had to define how rigs, coordinate systems, rotations, sequence lengths, and metadata would be interpreted and represented for learning-based models.

I started the Motion Capture Processor (MCP) by designing and implementing the initial package, then continued maintaining it from 2023 through 2026 as asset formats, rig conventions, model representations, and product requirements evolved. MCP established a reusable boundary between animation-domain assets and model-facing data: it loads and interprets motion, applies the required transformations and validation, and encodes the result as training-ready tensors.

The package grew with contributions from other AI Lab team members. My role remained focused on maintaining the shared architecture, evolving its representations and compatibility, and keeping the path from source assets to model inputs reliable as new use cases were added.

  • Ownership: Initial package architecture and implementation, followed by continued maintenance and evolution
  • Problem: No existing tool for converting production 3D motion assets into model-ready tensors
  • Inputs: Heterogeneous FBX and GLB motion assets and supported intermediate motion formats
  • Outputs: Validated tensor representations, sequence lengths and masks, metadata, and reproducible dataset splits
  • Package scope: Motion I/O, rig representation, normalization, canonicalization, retargeting, validation, tensor encoding and decoding, and dataset preparation
  • Collaboration: Researchers and engineers within the AI Lab
  • Period: 2023–2026

Product Problem

A production 3D asset is not yet a machine-learning sample. Files that appear to contain similar motion can use different skeleton hierarchies, joint orientations, root conventions, coordinate systems, frame rates, body proportions, or metadata. Those differences change the meaning of the data when it is encoded numerically.

Because no existing tool provided this conversion, model development could not begin with a reliable data representation. The lab first needed a consistent path from 3D assets to model inputs, including shared definitions for coordinate frames, joint ordering, rotation conventions, and variable-length sequences. Establishing this foundation made subsequent training experiments and model comparisons possible.

The package therefore needed to provide a consistent answer to four questions:

  1. Can the asset and its rig be interpreted without ambiguity?
  2. Can the motion be normalized or retargeted into the representation required by a model?
  3. Does the processed result satisfy the structural and numerical checks required for training?
  4. Can the tensor be decoded and inspected again in the 3D domain?

From 3D Assets to Training Tensors

flowchart TD
    A["3D Motion Assets<br/>(FBX, GLB)"] --> B["Parse Assets & Interpret Rig"]
    B --> C["Normalize · Canonicalize · Retarget"]
    C --> D["Validate Motion & Metadata"]
    D --> E["Encode Model Representation"]
    E --> F["Training Batch<br/>Tensor (B, T, J, C) · Lengths · Masks"]
    F --> G["Model Training & Evaluation"]
    F --> H["Decode to 3D Motion"]
    H --> D

MCP separates asset interpretation, domain-specific transformations, validation, and model-facing representation into explicit stages. This prevents file-format and rig logic from leaking into every training project and makes each transformation easier to inspect and test.

The reverse path is equally important. A model-facing tensor can be decoded into articulated motion so that changes to a representation can be checked against the 3D behavior they are intended to preserve. This round trip connects numerical model inputs with animation-domain review.

Package Design

The package was organized around responsibilities that could evolve independently while sharing the same motion and rig definitions:

  • Motion I/O and rig interpretation: Load source assets and reconstruct their joint hierarchy, rest pose, transforms, frame rate, and metadata
  • Shared motion representation: Provide a common in-memory form for offsets, rotations, rig information, global transforms, and sequence operations
  • Conversion and retargeting: Normalize coordinate and root conventions, canonicalize motion, convert rig bases, and retarget between supported body structures
  • Preprocessing and validation: Filter metadata, prepare assets, detect invalid samples, split datasets, and report processing outcomes
  • Model representation: Encode motion into batch-first Torch tensors and decode those tensors back into articulated motion
  • Tools and integration: Expose reusable package APIs and command-line utilities for dataset jobs, conversion, inspection, and rendering

The package boundary remained separate from workflow orchestration. Surrounding data jobs used Dagster or Airflow at different stages, but MCP kept the 3D-domain logic and model-facing contracts independent of either system. This allowed the execution environment to change without requiring the core conversion package to be rewritten.

A Tensor Contract for Motion Models

MCP bridges animation-oriented motion and a batch-first learning representation with shape (B, T, J, C):

  • B — batch: Multiple motion sequences processed together
  • T — time: Padded frames with explicit sequence lengths and masks
  • J — joints: The ordered joints defined by the target rig or body representation
  • C — channels: The per-joint representation, such as three coordinates for position, four values for a quaternion, or six values for a 6D rotation representation

The representation can be configured around the needs of a task rather than forcing every model into one fixed channel definition. Positions use three-dimensional coordinates, while rotations can use quaternion or 6D representations. Options also include local, root-relative, or global joint spaces; absolute or relative root motion; and task-specific velocity, contact, or temporal-delta signals. Canonicalization and normalization can be applied at the encoding boundary, while padding values and masks preserve variable-length sequences.

The same specification controls decoding. This keeps joint ordering, channel semantics, rotation conversion, normalization, and padding behavior consistent between training and reconstruction instead of duplicating that logic in model repositories.

Quality and Reliability

Validation is part of conversion rather than a cleanup step after training. The package rejects or records assets that cannot satisfy the expected rig and representation contract before those samples enter a dataset.

The validation path includes:

  • Rig hierarchy, joint ordering, and rest-pose compatibility checks
  • NaN, malformed-value, and invalid frame-range detection
  • Coordinate, root, and initial-pose canonicalization
  • Retargeting and representation round-trip tests
  • Deterministic train, validation, and test splits
  • Traceable metadata, processing reports, and failure reasons
  • 3D reconstruction or rendering for problems that are easier to see than to express as scalar checks

Numerical checks alone cannot reveal every motion problem. Reconstructing processed data through Blender and bpy made facing-direction errors, broken retargeting, misplaced roots, discontinuities, and other animation-visible issues easier to review in the domain where the data would ultimately be used.

Parallel and Resumable Processing

Processing a motion collection combines file I/O, CPU work, and Blender-dependent operations. MCP uses Ray to distribute independent assets while bounding concurrent work, reporting progress, and preserving completed outputs when a job is interrupted.

The dataset workflow supports:

  • Parallel processing with controlled concurrency
  • Resume behavior that skips completed preparation and conversion work
  • Reusable prepared assets across multiple dataset builds
  • Deterministic dataset partitions
  • Structured reports for completed, skipped, and failed items
  • Consistent package APIs that can be invoked from local tools or external orchestrators

These behaviors made large preprocessing jobs easier to inspect and restart without coupling the package to one scheduler or one model codebase.

Evolution and Maintenance

The first version established the path from source motion assets to learning-oriented preprocessing and tensor representations. I continued maintaining that foundation as the motion stack expanded, while other AI Lab team members added features and domain-specific extensions through the same shared package.

Maintenance across the package lifecycle included:

  • Supporting additional source and target rigs, body representations, and retargeting paths
  • Evolving root- and pelvis-based motion conventions while preserving compatibility with existing data
  • Adding SMPL-X-oriented conversion and representation support
  • Expanding Torch-based motion operations and configurable tensor codecs
  • Improving package installation, dependency compatibility, resource loading, and command-line tools
  • Adding tests for conversion parity, representation encoding and decoding, retargeting round trips, rendering, and preprocessing behavior
  • Deprecating or replacing earlier paths as the shared representation became more consistent

This was ongoing package work rather than a one-time preprocessing script. The maintenance goal was to let new formats and model requirements enter through explicit interfaces without forcing existing consumers to rebuild their data layer.

Adoption and Impact

MCP became the shared data foundation used across all motion and 3D model development within the AI Lab instead of remaining tied to its first use case. These model workflows could build on the same asset interpretation, transformation, validation, and representation rules without creating separate data-processing paths for each task.

The practical impact was a more consistent handoff between 3D assets and model development:

  • Researchers could consume normalized tensors without reimplementing 3D asset parsing and rig logic for each experiment
  • Model variants could share sequence, masking, rotation, and normalization behavior
  • Processed samples remained traceable to their source assets and conversion metadata
  • Representation changes could be decoded and reviewed again as 3D motion
  • New package contributions could extend a shared foundation instead of creating separate conversion stacks

Technical Summary

  • 3D asset processing: FBX, GLB, motion I/O, metadata filtering, normalization, canonicalization, and dataset preparation
  • Motion and rig representations: Skeleton hierarchy, rest pose, root- and pelvis-based rigs, coordinate spaces, joint transforms, and SMPL-X-oriented formats
  • Training tensor contract: Batch-first (B, T, J, C) tensors, explicit joint and channel dimensions, variable-length sequences, lengths, masks, configurable position and rotation representations, and normalization
  • Retargeting and validation: Rig conversion, cross-body retargeting, structural and numerical checks, round-trip reconstruction, and Blender (bpy) inspection
  • Parallel and resumable processing: Ray-based execution, bounded concurrency, resume behavior, deterministic dataset splits, and processing reports
  • Package maintenance and adoption: Shared Python package APIs, CLI tools, dependency and compatibility maintenance, test coverage, and reuse across all motion and 3D model development within the AI Lab