Finger Tracker

This page documents the Finger Tracker service provided with GML. The system and algorithms used are described in length in the paper Visual Tracking of Bare Fingers for Interactive Surfaces

shot.jpg

This service emits the position of the fingertips present in the camera view in real time, in the form of appearance, movement, and disappearance events.

General information about GML services is available on the parent page

Usage constraints

The closer the environment is to an "ideal" one, the more robust the system will be (i.e. it will make few errors, there will be less jitter, and latency will be lower).

field of view

The default service setting (see the service configuration panel) is 1.5mm per camera pixel, which corresponds to a 480x360 mm interactive surface. The tracker will work correctly within 1.0mm/pixel and 2.5mm/pixel, with similar performance.

This setting can be roughly approximated.

lighting conditions

Ideally lighting is mixed between office and sunlight. More importantly it is perfectly stable.

In practice, a moderate amount of variation is tolerable (such as created by clouds passing in front of the sun, or sunlight changing between the morning and the afternoon).

background

Even though parasite objects (coffee cups, pens) will not distract the tracker, it is recommended to keep the interactive surface as uniform as possible.

Background adaptation is disabled in the service. It is not robust enough to allow corect detection beyond a few minutes when the scene is modified substantially.

projector contrast

Projected information is actually background information for the tracker; since it typically varies greatly, it should be kept hidden from the camera.

Keep the projector contrast low enough to reach a 1:1 ratio between scene illumination by the projector and by the ambient lighting. Robustness generally increases when the camera is slightly over-exposed (enough to make the projected information disappear while keeping the users' hands visible).

performance
On a modern computer (CPU over 1GHz), the tracker should provide usable results (25Hz processing, ca. 80ms overall latency).

Programs

Below is a list of the programs provided with the GML that create a Finger Tracker service instance.

GML-Monolith-Simplified-FingerTracker
Launches a Frame Grabber service, a Simplified Calibrator service, and the Finger Tracker service. Connection between the three is performed automatically at launch.
GML-Monolith-FingerTracker
Launches a Frame Grabber service, a Automatic Calibrator service, and the Finger Tracker service. Connection between the three is performed automatically at launch.
GML-Component-FingerTracker
Launches only a Finger Tracker service instance.
GML-Demo-FingerTracker-Simple
This sample program connects to a Finger Tracker service and a Simplified Calibrator. Its interface displays colored blobs over the users' fingertips.
GML-Demo-FingerTracker-Basic
This sample program connects to a Finger Tracker service and a Automatic Calibrator. Its interface displays colored blobs over the users' fingertips.
GML-Demo-Shuffler
This sample program connects to a Finger Tracker service and a Automatic Calibrator. Its interface displays a set of photographs that can be translated, scaled and rotated using one or more fingers.

Communicating with the service

To start receiving events from the service, you must

  1. Open a TCP connection to the relevant service port.
  2. Send an empty message to the service.
  3. Start sending and receiving messages.

Messages are formatted as plain ASCII text, with carriage return plus line feed line termination (DOS convention, crlf). All messages are concluded by an empty line, i.e. by the crlf sequence. Each message consists of one or more lines, the first of which is a header line that can usually be ignored safely upon reception. It consists of:

  • the magic word BIP/1.0;
  • a unique ID for the emitting peer (e.g. a random number), formated as an 8-digit hexadecimal number;
  • the index of the message since the connection started (starting from zero), using the same format;
  • the length, in bytes, of the message body, excluding the line terminator immediately following the header, the the line terminator that concludes the message, using the same format.

Two example messages:

BIP/1.0 A47F64A1 00000000 0000000

BIP/1.0 A47F64A1 00000001 000000D
hello, world!

The header line is followed by zero or more events, one per line. Each event consists of a name, written between angle brackets, and zero or more pieces of meta-information, separated by space.

Example message containing an event:

BIP/1.0 A47F64A1 00000A14 0000025
<Motion> 173 172.15 240.36 1319768891

For more details and communication using UDP or multicast UDP, refer to the BIP specification.

Channels

The finger tracker will only run (i.e. start emitting tracking events) if both the calibration and grabber channels are linked to a peer, and the calibration matrix is known.

Note that in the monolithic version of the finger tracker, this is done automatically.

events (output)

The the events channel emits the fingertip tracking events (tm is always the current time in nanoseconds).

<Detect> x y ox oy r tm
a fingertip candidate has been detected at (x,y), with orientation (ox,oy) and radius r.
<Appear> oid x y ox oy r tm
a finger has appeared at x,y; oid is unique to this finger.
<Motion> oid x y ox oy r tm.
finger oid has moved to x,y
<Dwell> oid x y ox oy r tm.
finger oid was static at x,y for over the parameter dwell delay.
<Disappear> oid x y ox oy r tm.
finger <oid> has disappeared at x,y.

Other events may be emitted but should be ignored.

calibration (inoutput)

This channel is used to query and receive a calibration matrix from a calibration service, using either the simplified protocol or the standard protocol (see the documentation for the calibration service).

The link will be established automatically.

grabber (input)

Used to maintain the connection to the source of images in which fingertips are tracked. The link will be established automatically.