MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching

§ Abstract

§ Contents

1 Introduction & Motivation

Human motion is fundamentally governed by the neuromuscular system, in which observable kinematics emerge from complex coordination patterns of muscle activation. Due to the redundancy of the musculoskeletal system, visually similar movements can arise from substantially different internal control strategies — some of which may compromise safety and performance. Existing Action Quality Assessment (AQA) research remains limited: it primarily concentrates on exteroceptive visual inputs, struggles to capture implicit physiological states, treats actions as monolithic patterns, and relies on datasets with inconsistent quality and coarse-grained annotations.

Core Contributions

• 📊
  MyoMechanix Dataset 7,500+ samples across 20 weight-loaded actions from 38 subjects — synchronized 5-view RGB video, 3D pose, sEMG, heart rate & respiratory rate. The largest multimodal AQA benchmark.
• 🧠
  Fitness Knowledge Graph (FKG) Formalizes actions as structured representations — decomposing each into hierarchical phases, ordered key steps (AKs), error types (ETs), and corrective feedback (FBs). Enables compositional, penalty-based scoring.
• 🎯
  CUBIST Model Compositional Ontological Reasoning Engine — performs decomposition→analysis→recomposition over structured actions, producing interpretable error attribution and corrective feedback.
• 🏆
  Three Benchmarks MyoMechanix-AQA (4 split protocols), MyoMechanix-VideoQA (30,048 QA pairs), and Video2EMG — a novel cross-modal task predicting muscle activity from video.

★ System Overview

Figure 1. An overview of our MyoMechanix ecosystem. We advance fitness activity understanding along two complementary dimensions: physical fidelity (multimodal sensing with sEMG) and semantic richness (FKG-structured representations).

2 Related Work

We comprehensively compare MyoMechanix with prior art across four areas: Action Quality Assessment Datasets & Models, Video Question Answering Datasets, Electromyography Datasets, and Muscle Activity Estimation. Existing AQA datasets suffer from: lack of physiological signals, insufficient diversity in ability levels, lack of intermediate structured reasoning, and a scientific blind spot in load-based movement modeling.

3 MyoMechanix Dataset

Design Principles

MyoMechanix focuses on weight-loaded compound exercises — 20 representative actions selected to cover both upper- and lower-body muscle groups, involve complex and injury-prone joints (shoulders, knees, hips), include free-weight actions requiring stabilization, and represent widely practiced exercises. 10 barbell + 10 dumbbell exercises ensure balanced evaluation across equipment types.

Figure 2. The 20 representative weight-loaded actions collected in MyoMechanix.

Multimodal Data Collection

Fitness Knowledge Graph (FKG)

The FKG provides an ontology-like representation that formalizes weight-training exercises as biomechanically phased action representations. Each exercise is decomposed into three phases — Preparation, Concentric, and Eccentric — with ordered action key steps (AKs), phase-conditioned error types (ETs), and corresponding corrective feedback (FBs). Error severity is hierarchically weighted: joint safety (9–10), core stability (7–9), foundational posture (6–7), fine control (5), breathing (2).

Figure 3. Entity relationships within the Fitness Knowledge Graph — mapping actions → key steps → error types.

Compositional Scoring

The overall score is computed bottom-up: S = [1 − ΣW_observed / ΣW_max] × 100. This system separates mathematical complexity from annotation, ensuring both rapid labeling and full interpretability — in stark contrast to opaque holistic scoring approaches.

Figure 4. Compositional scoring process for Barbell Overhead Press — individual errors map to score deductions, yielding a final transparent score of 77.02.

Annotation Statistics

4 CUBIST Modeling Paradigm

CUBIST (Compositional Ontological Reasoning Engine) is inspired by the Cubist art movement's principle of decomposing complex objects, examining each part from multiple viewpoints, and recombining them to reveal underlying structure. It moves beyond monolithic black-box scoring toward interpretable, step-wise reasoning that closely reflects expert analysis.

Figure 5. CUBIST model architecture (multimodal version) for the AQA task.

Three-Stage Progressive Design

Stage I — Action Perception-driven Expert Routing: A spatiotemporal video encoder with attention-based pooling classifies the action and routes the input to the appropriate action-specific expert network. Built on a ViT backbone with trilinear-interpolated positional encoding for up to 103-frame videos.

Stage II — Ontology-driven Error Reasoning: A temporal adapter recovers native spatiotemporal topology, an implicit phase parser discovers action phases without manual annotations, and a phase-aware error query decoder uses Mixture-of-Experts (MoE) architecture to predict fine-grained error probabilities. Asymmetric loss handles the high-dimensional, sparse multi-label nature of error detection.

Stage III — Compositional Scoring Engine: A deterministic neuro-symbolic module that applies annotation-defined error weights and the compositional scoring formula — introducing zero additional learnable parameters. The final score is directly attributable to interpretable error concepts, improving transparency at both reasoning and output levels.

Multimodal Fusion Strategy

CUBIST employs a feature-disentangled late-fusion design that avoids prematurely forcing heterogeneous modalities (RGB, 3D Pose, sEMG) into a shared representation space. Each modality develops task-relevant representations independently before integration, preserving modality-specific discriminative characteristics while reducing cross-modal gradient interference.

5 Experiments & Results

Three Benchmark Tasks

AQA Results — Vanilla Split

CUBIST achieves new SOTA on unimodal RGB evaluation.

AQA Results — Multimodal Fusion

Progressive multimodal fusion yields consistent gains. CUBIST(MV+Pose+EMG): +18.91% ρ over DAE, +12.86% over unimodal CUBIST.

Figure 6. Qualitative comparison of CUBIST against baseline methods on the AQA task.

VideoQA Results

Fine-tuning Qwen2.5-VL-3B (LoRA) achieves substantial gains: BLEU 0.0415 → 0.1970, CHRF++ 39.53 → 52.00. However, score prediction accuracy remains challenging — R-ℓ₂ error decreases by less than 2%, suggesting that numerical reasoning in VLMs remains a significant open problem. Current VLMs still have ~50% room for improvement in text generation and up to 80% in score prediction.

Video2EMG Results

The best configuration (ViT + SVR) achieves MAE = 0.1465, RMSE = 0.1882, CCP = 0.4345 (moderate waveform alignment). Performance degrades significantly under cross-subject and cross-view protocols, highlighting the difficulty of cross-modal physiological inference — leaving approximately 56% room for future improvement.

Figure 7. Qualitative results of Video2EMG — predicted vs. ground truth sEMG waveforms.

6 Conclusion

We introduced MyoMechanix, a biomechanically grounded multimodal ecosystem for skilled activity understanding and coaching, establishing a compositional paradigm for action quality assessment that moves beyond purely visual, monolithic action modeling. Three key findings: (i) effective AQA benefits from integrating latent biomechanical signals with visible motion cues; (ii) language-grounded reasoning over structured action knowledge enhances fine-grained VLM understanding; and (iii) promising Video2EMG results suggest a path toward estimating muscle dynamics from accessible visual observations. Despite these advances, MyoMechanix remains a challenging benchmark, motivating future research in cross-subject generalization, richer causal biomechanical modeling, and integration with embodied systems.

Acknowledgement

Supported by the Suzhou Basic Scientific Research Project (SSD2024013), the Doctoral Student Program of the Young S&T Talents Cultivation Project, CAST. Fast Move Technology is especially acknowledged for providing sEMG devices.

§ BibTeX Citation

If you find this work useful, please cite:

@article{yin2026myomechanix,
  title     = {MyoMechanix: Biomechanically-Grounded Compositional
               Skilled Activity Understanding and Coaching},
  author    = {Yin, Hao and Parmar, Paritosh and Gu, Lijun and
               Xu, Lin and Guo, Tianxiao and Liu, Xiujin and
               Zheng, Tianyou and Zhang, Yang and Fu, Weiwei},
  journal   = {Under Review},
  year      = {2026},
  url       = {https://github.com/HaoYin116/MyoMechanix}
}