Pose Detection

Body and head tracking with a normal webcam. It runs on the device itself, for games, exhibits and accessible controls.

Pose detection finds the key points of a body or face in each camera frame. Key points are fixed spots the computer can find again in every frame. We use them to turn movement into input: a turn of the head, a raised arm, running on the spot. It works with an ordinary webcam. It runs in the browser or app on the device itself. So no video has to leave the machine.

We have built it into several of our own projects. Jet-eye is a shooting game you control with head movement and blinks. It was made for people who cannot use their hands. Our interactive kiosk is a stand-alone screen in a public place. It has a body-test mode that draws only the skeleton. This mode scores a standing pose or a horse-stance pose (a martial-arts squat). In another kiosk mode, flapping your arms makes migrating geese fly faster. A head-tracking demo shifts a 3D scene as you move your head. This makes the screen look like a window. A running prototype (an early test version) reads your running cadence, or step rate, from a webcam. It uses that to set the playback speed of a course video.

The hard part is usually not the model. It is the details around it. The points jitter (shake slightly) even when a person stands still. Cameras can be slow to start. Settings need to fit each user. Our projects record these lessons and the fixes.

Gallery