Brief us
Docs

Formats, schemas, loaders

Everything ships in formats your pipeline already reads. The schemas below are the exact contracts that ship with deliveries.

Annotation contracts

One JSON per record

Formally specified with JSON Schema. These are the delivery-record contracts; the live sample viewer on this site serves a reduced, web-safe subset of the same fields.

textphotos.schema.json

Image dimensions, language and scene labels, text-region polygons, transcripts and annotation review state.

 annotation.schema.json

Egocentric action annotation: viewpoint, domain, steps, segments, EPIC-100 taxonomy, word-level ASR.

 th.schema.json

Talking-head records: face track, landmarks, phoneme→viseme timeline, emotion, AV-sync and explicit per-gate quality verdicts.

 motion.schema.json

Human-motion records: curation labels, rights with attribution line, per-file hashes, the per-frame hand-reliability flag with its measured residual, and the shapes of the 133-keypoint 2D and 3D pose sidecars.

 annotation.example.json

A canonical worked example of the egocentric contract.

 
{
  "clip_id": "…", "viewpoint": "head_cam", "egocentric_score": 0.93,
  "domain": "cooking", "duration_s": 45.0, "width": 3840, "height": 2160,
  "caption": "A person kneads dough on a floured counter…",
  "asr":      [ { "start": 0.4, "end": 3.1, "text": "…",
                  "words": [{ "w": "knead", "s": 1.1, "e": 1.4 }] } ],
  "steps":    [ { "step_index": 1, "start": 0.0, "end": 34.2,
                  "summary": "Knead the dough" } ],
  "segments": [ {
    "start": 3.2, "end": 11.8, "action_verb": "knead",
    "taxonomy": { "epic_verb": "knead", "epic_noun": "dough", "sim": 0.91 },
    "caption": "Two hands press and fold pale dough on a wooden surface.",
    "asr_text": "now we knead the dough for about ten minutes"
  } ]
}
Delivery

Formats

FormatTypical use
WebDataset (.tar shards)media + JSON sidecars, streaming-friendly training input
LeRobotDataset v3robot-learning stacks — export prepared per delivery
RLDS / GR00T exportembodied-AI frameworks expecting episode structure — per delivery
Parquet manifestmetadata queries, filtering, joins before you touch media
S3 / GCS transferdelivery into your bucket and region

When you ask for splits, they are cut by source video ID — never by clip — so near-duplicate frames cannot straddle train/val.

Loaders

pip install → training batch

webdataset 
import webdataset as wds

ds = (
    wds.WebDataset("egocentric-{000000..000042}.tar")
    .decode(wds.torch_video)
    .to_tuple("mp4", "json")
)
for video, annotation in ds:
    print(annotation["clip_id"], len(annotation["segments"]))
huggingface datasets 
from datasets import load_dataset

ds = load_dataset("parquet", data_files="manifest.parquet")["train"]
row = ds[0]
print(row["clip_id"], row["domain"], row["license"])
Provenance manifest

One line per record

Illustrative row (identifiers redacted). Actual manifests carry the full source URL, exact license and hash chain for every record.

{"clip_id": "a1b2c3d4e5f_120000_165000",
 "clip_path": "shards/egocentric-000007.tar#a1b2c3d4e5f_120000_165000.mp4",
 "domain": "cooking", "viewpoint": "head_cam", "egocentric_score": 0.93,
 "manipulation_density": {"score": 0.74, "hands_frac_clip": 0.81},
 "rights": {"source_license": "documented", "leakage_clean": true},
 "provenance": {"source_url": "<recorded>", "fetched_at": "2026-07-14T09:31:02Z",
                "transform_hash": "sha256:9f2c…"}}