Skip to content

Project Foresight

Robotics • May 13, 2026

JEPA-style visual world model for an SO-101 leader/follower robot arm. It learns normal camera-and-action dynamics, then flags unexpected physical events live.

Robotics
World Models
JEPA
Computer Vision
Python
LeRobot
Anomaly Detection
Hardware

Key Features

  • •SO-101 leader/follower teleoperation running at 60 Hz
  • •Synchronized USB camera, action, target, observation, and error logging
  • •JEPA-style action-conditioned predictor trained on future visual embeddings
  • •Live anomaly monitor with session threshold calibration
  • •Recorded anomaly evaluation for hand intrusion, object intrusion, occlusion, and unexpected object movement
  • •Short-horizon H6 checkpoint for about 0.1 second live responsiveness
  • •Guarded autonomy prototype that samples actions against a learned goal embedding
  • •Human annotation tooling for pick-and-place event labels

Project Foresight is a robotics prototype for giving a physical SO-101 arm a learned sense of what should happen next. The system watches the robot through a USB camera, conditions predictions on the follower target actions, and compares predicted future embeddings against the real future camera stream.

When the prediction error spikes, the live monitor flags the scene as anomalous. In the current demo, that means detecting obvious physical surprises like a hand entering the workspace, an object moving unexpectedly, or the camera being occluded.

Demo

Watch the project demo on YouTube: youtu.be/q240DAB6ssE

System Loop

The prototype connects the full robotics learning path:

  1. SO-101 leader/follower teleoperation
  2. Camera and motor telemetry recording
  3. JEPA-style visual/action world-model training
  4. Live prediction-error anomaly detection
  5. First guarded autonomy scaffold for goal-directed planning

The current model predicts future visual embeddings instead of pixels. That keeps the live loop lightweight enough to run alongside teleoperation while still producing a useful anomaly score.

Current Status

The V1 demo uses a fixed external camera rather than a wrist-mounted camera. It is a working prototype for the current room, camera pose, and task distribution, not a general robot safety system yet.

The next milestone is turning warning-only anomaly detection into a conservative halt behavior after consecutive anomalous frames.

Repository: github.com/tylergibbs1/project-foresight-so101