+7
日增长
1,003
stars
107
forks
27
watchers
215
prs
7
issues
4
releases
README
Video to Data (V2D) An end-to-end pipeline that converts human demonstration videos into simulation-ready assets and physics-grounded robot training data. Documentation · Datasets (Hugging Face) · Robotic Grounding Project Page · Robotic Grounding Tech Report !Video to Data pipeline — from human demonstration video through ingestion, reconstruction, and robotic grounding in Isaac Lab to a physics-grounded policy, dataset, and real-robot deployment --- Contents - Overview - E2E workflow - Packages - Prerequisites - Quickstart - Video Ingestion Agent - Reconstruction - Robotic Grounding - FlashCHORD accelerated training - Design philosophy - Contributing --- Overview Video to Data (V2D) turns raw human demonstrations into robot-ready training data through three composable stages and a repository-root E2E workflow. Each stage runs independently and writes its artifacts to disk, so you can stop, inspect, cache, and recompose the pipeline at any boundary. 1. Video Ingestion Agent — a LangGraph-driven agentic workflow that segments demonstration videos into temporally-bounded action clips, extracts an entity-relation scene graph, and stores per-frame SigLIP-2 embeddings. The result is a queryable action database (graph.db + vector.db) that lets downstream stages select which clips to process via natural-language retrieval, instead of brute-forcing the full video. 2. Reconstruction — containerized vision modules turn the selected RGB (or stereo) clips into per-frame depth, object masks, textured meshes, 6-DoF object poses, and SMPL human body parameters. Multi-view pipelines (run_mv_hoi_reconstruction, run_mv_calibration) orchestrate the full reconstruction from a rosbag. 3. Robotic Grounding — human motion (e.g. Arctic) is retargeted onto the target robot embodiment (Sharpa), then the reconstructed scene and retargeted motion drive Isaac Lab environments trained with RSL-RL PPO to produce deployable policies. E2E workflow An end-to-end workflow serves as a reference for connecting the repository's components into a complete reconstruction-to-policy pipeline. The included example starts with a raw egocentric human demonstration captured on an iPhone and continues through reconstruction, retargeting, source-expert training, demonstration collection, GR00T N1.7 fine-tuning, and closed-loop evaluation. It has been verified with both floating Sharpa and Vega Sharpa embodiments. run_e2e.sh provides a reproducible launcher that orchestrates the stages and records their state for inspection and resumption. A companion agentic skill uses the launcher under the hood to guide setup and execution. The main path trains its own source expert; the bundled tissue-box ONNX policy is an optional shortcut for users who want to start directly with GR00T fine-tuning. The complete workflow can run on a single NVIDIA RTX A6000 GPU with 48 GB of VRAM and has also been verified on NVIDIA L40 and L40S GPUs. Follow the E2E workflow guide for the runnabl
Top issues
Latest release
Release v0.4.0 v0.4.0 2026-09-23Highlights V2D v0.4.0 introduces FlashCHORD for accelerated GPU-parallel robotic reference-tracking training for loco-manipulation and dexterous manipulation, alongside improvements to reconstruction tools, robotic grounding, and closed-loop GR00T evaluation. FlashCHORD (NEW!) Accelerated policy training from human-object motion using Newton, Warp, and JAX. Summary - Training recipes: adds FlashSAC and PPO training in Newton for floating Sharpa, Vega Sharpa, and G1 + Dex3. - Complete pipeline: documented setup, robot and scene inspection, kinematic and dynamic reference rep