Pose Detection

Webcam-based body and head tracking that runs on the device, for games, exhibits and accessible controls.

Pose detection finds the key points of a body or face in each camera frame. We use it to turn movement into input: a turn of the head, a raised arm, running on the spot. It works with an ordinary webcam and runs in the browser or app on the device itself, so no video has to leave the machine.

We have built it into several of our own projects. Jet-eye is a shooting game controlled by head movement and blinks, made for people who cannot use their hands. Our interactive kiosk includes a body-test mode that draws only the skeleton and scores a standing or horse-stance pose, and a mode where flapping your arms makes migrating geese fly faster. A head-tracking demo shifts a 3D scene as you move your head, so the screen looks like a window. A running prototype reads running cadence from a webcam to set the playback speed of a course video.

The hard part is usually not the model but the details around it: points jitter even when a person stands still, cameras can be slow to start, and settings need to fit each user. Our projects record these lessons and the fixes.

Gallery