Hand tracking

Pick a backend (WiLoR, MediaPipe, HaMeR), get 21-joint hand poses anchored to depth, and stash MANO outputs for HDF5 export.

HandTracker is a unified wrapper around three backends. They all return the same HandPose schema, so the rest of your loop is backend-agnostic:

from stera.models import HandTracker

tracker = HandTracker(model="mediapipe")  # or "wilor", or "hamer"
for frame in session.frames():
    hands = tracker.detect_hands(frame)   # list[HandPose]

Picking a backend

All three anchor 3D joint positions to the depth image when frame.depth is present, so the same loop works whether you're running CPU MediaPipe or GPU WiLoR.

MediaPipe

from stera.models import HandTracker

tracker = HandTracker(
    model="mediapipe",
    max_num_hands=2,
    min_detection_confidence=0.6,   # tunable, see Configuration below
)

for frame in session.frames():
    hands = tracker.detect_hands(frame)
    for hp in hands:
        print(hp.hand_side, hp.wrist.x, hp.wrist.y, hp.wrist.z)

The MediaPipe asset (~6 MB) is downloaded to ~/.cache/mediapipe/hand_landmarker.task on first call.

WiLoR

tracker = HandTracker(
    model="wilor",
    model_path="/opt/WiLoR",       # local clone of rolpotamias/WiLoR
    save_mano_vertices=True,       # default; needed for /hand-pose MANO export
)

for frame in session.frames():
    hands = tracker.detect_hands(frame)

WiLoR runs YOLO + ViTDet on each frame, then a MANO regressor on each hand bbox. The full MANO outputs (778×3 vertices, 10 betas, global orient, per-finger pose) are stashed onto each HandPose and written to annotation.hdf5:/hand-pose by session.export.

See Installation → WiLoR.

HaMeR

tracker = HandTracker(
    model="hamer",
    model_path="/opt/hamer",       # local clone of geopavlakos/hamer
)

HaMeR's pipeline:

1. detectron2 (Cascade Mask R-CNN ViTDet-H or RegNet-Y) → human bboxes.
2. ViTPose+-Huge wholebody → 133 keypoints; the last 42 = both hands.
3. Build hand bboxes from confident hand keypoints.
4. HaMeR → MANO regression on each hand bbox.

Same HandPose outputs as WiLoR, same MANO extras. First call downloads ~3 GB of detector weights from FAIR public files; everything is cached after that.

What detect_hands accepts

tracker.detect_hands(frame)                      # SyncedFrame → uses frame.rgb / .depth / .depth_K
tracker.detect_hands(rgb)                        # raw (H, W, 3) uint8 array
tracker.detect_hands(rgb, depth=d, intrinsics=K) # both explicit

When you pass a SyncedFrame, the wrapper plumbs frame.rgb, frame.depth, and frame.depth_K (or frame.rgb_K as fallback) for you. Pass raw arrays only when you've decoupled the loop from MCAPReader.

What you get back

hands: list[HandPose]

for hp in hands:
    hp.hand_side             # "left" or "right"
    hp.confidence            # float
    hp.wrist                 # Keypoint
    hp.fingers["index"][3]   # tip of the index finger
    hp.all_keypoints         # flat list of 21 Keypoints (or [wrist] for wrist-only)
    hp.has_fingers           # True if full hand was returned

See HandPose for the full schema.

For backends that produce MANO (WiLoR, HaMeR), private _mano_* attributes are also stashed on the HandPose and forwarded to HDF5 export, you don't normally read them yourself.

Configuration

Pass any backend config field as a kwarg to HandTracker(...). The wrapper forwards it to the backend's Config dataclass.

MediaPipe

WiLoR

HaMeR

All three backends also expose sanity-check thresholds (max_joint_abs, min_wrist_depth, min_palm_span, max_palm_span), bumps with implausible joint configurations are dropped silently.

Patterns

Attach to the session for HDF5 export

for frame in session.frames():
    hands = tracker.detect_hands(frame)
    session.add_hand_pose(frame.index, hands)

session.export("episodes/run_01")

The accumulated detections land in annotation.hdf5:/hand-pose as (F, 21, 3) arrays per side. See HDF5 schema.

Visualize over RGB

from stera.viz import Visualizer
viz = Visualizer(session)

for frame in session.frames():
    hands = tracker.detect_hands(frame)
    viz.log_frame(frame, hands=hands)

viz.export("hands.rrd")

Hands appear as 2D overlays on the RGB panel and as 3D Points3D in the world scene when a camera pose is present.

When 3D isn't available

If your recording has no depth, joints come back with z=0 and Keypoint.x/y carry pixel coords. HandPose._kpts_2d_rgb is also stashed as a (21, 2) array.

See also

• HandTracker API, constructor and methods.
• HandPose, output schema.
• HDF5 schema → /hand-pose, what gets exported.