BIP/1.0: Basic Interconnection Protocol for Event Flow Services

Authors:	Julien Letessier Dominique Vaufreydaz Sebastien Pesnel Remi Emonet
Revision:	47
Status:	Draft
Date:	September 6, 2006

Abstract

This document describes low-level and high-level aspects of the BIP/1.0 protocol that allows the cooperation of services used to build interactive systems.

Contents

Introduction
Terminology
Services overview
Communicating with services
Inspection and control of services

Introduction

BIP is designed to meet a number of requirements encountered when constructing interactive systems:

function reuse and distribution

Subsystems may need to be used by multiple other subsystems, possibly running on different machines (for perfomance or geographic reasons).

For instance, a video capture service might be used concurrently by a surveillance system located on a centralized server, a video-conference application and a motion capture service, both co-located to the camera.

low threshold, high ceiling

Easy integration of BIP services into heterogeneous systems is a requirement. Minimalistic implementations (under 100 lines of code in any language) should always possible, in order to allow for easy interoperability.

performance

Interactive systems always place a constraint on overall latency.

For example, in the case of a finger tracker service used for interaction on an augmented surface (an example of tightly coupled interaction), the latency must be under 50 ms to maximize usability. On the other hand, for a service that counts the number of persons present in a room, a latency of 1 to 5 seconds might be acceptable.

robustness

In some cases, information needs to be reliably conveyed between subsystems, i.e. any message sent by a service must arrive to a listening peer (possibly within a tolerable time limit).

This may be the case, e.g. for the person counter described above.

These requirements have a number of consequences that influence how BIP is designed:

it is presumed that the systems communicating using BIP are isolated, black-box services;
no assumption is made on the peer operating systems, languages, or method of communication (machine-local or distant);
services must use user-friendly naming, because interconnection will be performed by humans in the general case;
peer services must be dynamicaly discoverable in order to alleviate the need for address-based and port-based setup;
multiple transports must be available to cope with the multiple latency and reliability needs;
the protocol must contain a minimal portion that is sufficient to make services useful, while allowing simple programs like telnet to interoperate with any BIP implementation.

The minimal BIP protocol only consists in an exchange of ASCII messages with 1-line message headers over a TCP socket. This allows for minimal interoperability with a very low effort. Implementors of BIP can enrich the inteconnection by adding support for UDP communications, multicast UDP, DNS-based dynamic discovery, and finally service control through an XML control protocol.

Terminology

Service: An opaque processing unit that accepts given input events and produces given output events. It possesses a number of state variables (variable values) and properties (constant values), altogether refered to as parameters. Interaction with a service is acheived by establishing a link with one or several of its channels.
Peer: An entity on the network that can communicate with a service: e.g. another service, an architecture adapter, or a user application.
Channel: A named point of communication between a service and one or several peers. One or more links can be established with a channel. There are three type of channels, depending on the direction of the flow of data: input channels, output channels, and duplex channels. Note that this is different from system-level sockets, e.g. TCP or UDP sockets.
Link: A means of sending messages to a service or receiving messages from a service. A link is comparable to an established socket-based network connection, except that no assumption is made on the underlying transport (TCP, UDP, multicast UDP).
Message: An encapsulation of one or several events for transmission over a link.
Event: A discrete piece of information that transits between a service and a connected peer.

Services overview

Discovering a service

BIP services advertize themselves using the DNS-SD¹ convention over Multicast DNS². They use the local. domain, TCP transport, and the bip service name; in other words, fully qualified names for BIP services names must be as follows:

<name>._bip._tcp.local.

where <name> is chosen by the service instanciator. The TCP port number advertized in the DNS-SD SRV record is the port used for control channel to the service (see below).

The DNS-SD TXT record fields listed below must or may be present for each service. They must not change during the lifetime of a service.

id : required: The peer id used to identify this service during communcations with other peers.
class : optional: The class of service. This will be standardized in a future revision of BIP. Classes starting with a period are reserved.
owner : optional: A descriptor of who owns the service. This can be used to limit connections to a cluster of services, e.g. in the case of concurrently running systems that uses the same services. It is advised to use either the login or the full name of the user who instantiated the service.

Additionally, there must exist a field for each channel. The field key is the channel name, and the field value is of the form port/type, where port is the channel's TCP port, as a decimal number, and type is one of i, o or d, for input channels, output channels, and duplex channels, respectively.

Example record:

id=FADA97CE
class=bip.source.noise
owner=mezis
events=123/d
noise=456/o

[1]	DNS-Based Service Discovery: browsing and discovery of services using DNS queries. Described at http://www.dns-sd.org/.

[2]	Multicast DNS: performing peer-to-peer DNS queries over IP Multicast. Described at http://www.multicastdns.org/.

Notes:

TXT record field labels (e.g. "inputs") are case-insensitive. It is recommended to only use lower-case letters, numbers, hyphens and underscores in labels.
Channel names (e.g. "events", "noise") should be short human-readable names. They must be formed only of lower-case letters, digits, hyphens and underscores.

Connecting to a service

There are four possible types of channels within a service:

the control channel vehicles events that allow a peer to inspect a service and control its state variables;
input channels transmit events to one of a service's inputs;
output channels transmit events from one of a service's outputs.
duplex channels are used for transactions between peers, for instance as a support for remote method invocation support.

A service accepts TCP link on one TCP port for control links, and one TCP port for each input and output. For inputs, it also accepts messages over UDP if a TCP link has been established (see below). For outputs, it can emit over UDP if requested.

Data that transits over any link with service obeys the message format described below (in Low-level message format). For the control link, data inside messages obeys further specification (see the sections on inspection and control below).

Service parameters

Services may define any number of parameters.

All services must define at least the following so-called "universal" parameters.

status : integer

Possible values are

the service is not running;
the service was asked to run, but is waiting for one or more inputs to be connected;
the service is running.

lock : integer

Possible values are

zero; or
the ID of a peer connected to the control port.

If non-zero, the service will not modify any parameter upon reception of a control query that originates from a peer which ID is not equal to the value of this parameter. A control answer event will still be emitted.

This parameter is automatically reset to zero if the corresponding peer disconnects.

Communicating with services

Motivation: the communication protocol must not impose any software requirements on the peers that connect to a service.

Low-level message format

Messages are the lowest-level of the protocol, i.e. what transits over a network connection to a service. They contain a fixed-size (34 bytes) header and a variable-size payload; the header is the concatenation of the following ASCII strings:

Magic header (7 bytes). It is a string of the form BIP/X.Y, where X is the major version of the protocol, Y is the minor version. All peers should emit BIP/1.0, and accept any form BIP/1.Y.
Space (1 byte).
Peer identifier (8 bytes). A unique identifier for the peer that emitted the message. It represents a 32-bit unsigned integer formatted as hex.

For peers that are services, the first 16 bits may represent the service startup time in seconds since the UNIX epoch, modulo 65536, and the last 16 bits may be randomly chosen³.
Space (1 byte).
Message identifier (8 bytes). A message identifier that is unique for the (simplex) link. It represents a 32-bit unsigned integer formatted as hex. It is incremented for each sent message and starts from 0. The message identifier 0 is used during the link establishment process (see Establishing a link).
Space (1 byte).
Payload size (8 bytes). The number of bytes in the message contents. It represents a 32-bit integer formatted as hex.

[3]	In order to prevent identifier collisions, a service should use the best random algorithm available. In particular, on UNIX platforms, the random number generators in the standard C library should be seeded, e.g. with the service startup time.

Both the header and the payload are immediately followed by the <cr> carriage-return plus <lf> line-feed line termination sequence (bytes 0x0A and 0x0D), which are not taken into account for length calculations. In the case of an empty payload (length zero), the termination sequence must still be appended; in other words, the message end with two successive <cr><lf> sequences.

Example message:

BIP/1.0 DEADBEEF 00000000 0000000<cr><lf><cr><lf>
BIP/1.0 DEADBEEF 00000000 000000D<cr><lf>hello, world!<cr><lf>

Note that the format of message contents is not specified for input, or output duplex links to a service. However, it is specified for the control connections (see the sections on inspection and control below).

It is strongly advised to keep the format of message contents for a particular service as simple as possible, for the sake of inter-operability with other systems and services. In particular, service designers should:

avoid using high-level languages except where relevant: XML is pertinent for large tree-like structures, not for transmiting two integers;
transmit plain ASCII text instead of binary when possible: it will be easier to trace the protocol for debugging purposes;
use <cr><lf> as a line delimiter even inside messages.

Establishing a link

A service must accept only one control link at a time. A service may choose to only accept one link to any input, output or inoutput channel at a time.

To establish a link to a service, a peer must:

Open a TCP connection to one of the service's ports (either the control port or one of the input, inoutput, or output ports);
Send an initial message to the service, with message number zero;
Receive an initial message from the service, with message number zero.

These two messages allow the peers to identify eachother, thanks to the peer ID provided with each message. The order of steps 2 and 3 is not specified. These initial messages may be empty or contain one or several lines of meta-information about the link, the channel, or the service.

For instance, suppose peer A with peer ID DEADBEEF opens a TCP connection to peer B with peer ID DADABABA. Peer A then sends the message:

BIP/1.0 DEADBEEF 00000000 00000000<cr><lf><cr><lf>

to peer B, and peer B sends the message:

BIP/1.0 DADABABA 00000000 00000000<cr><lf><cr><lf>

to peer A. The link is then considered to be established.

Link meta-information is formatted a la RFC822, in successive key-value pairs. Keys must be formed of lower-case letters, digits and hyphens. Values must be formed of printable 7-bit ASCII characters. Key and value are separated by a colon and any number of space or tab characters. Successive key-value pairs are separated by the <cr><lf> line delimiter. Example (delimiters ommited):

BIP/1.0 DEADBEEF 00000000 00000066
service-name:   Dummy Service
channel-name:   Dummy output channel
udp-port:       21587
status:         Up and running
content-type:   text/xml

A number of keys are reserved (though optional):

content-type is the MIME format of all messages. If not specified, the default if text/plain.
udp-port can be used to establish the UDP part of a link (see Communicating over UDP);
multicast-group can be used to establish the multicast UDP part of a link;
if present, service-name must be the service name;
if present, channel-name must be the channel name.

Communicating over TCP

Communicating over TCP is possible for any service link. Once a link has been established as described above, communication is acheived by sending and receiving any number of successive messages.

It is recommended that services that require low-latency communication over TCP send and receive messages (e.g. empty messages) at all times. This is to ensure that the TCP window stays as open as possible, and that occasional "bursts" of data are not limited by the enpoints' TCP stacks.

The actual minimal data throughput to maintain depends on you local implementation(s) of TCP; it is however recommended that you maintain an overall average throughput of half your maximal throughput between the two peers.

Communicating over UDP

Communicating over UDP is possible for any link (for service input, output and inoutput channels), except control links. A "source" peer may send messages over UDP to a "destination" peer if

a link has been established (over TCP) as described above; and
the destination UDP port is known. This is possible if the "destination" peer included the udp-port key in its initial message.

Each UDP datagram must contain exactly one message.

A service should ignore incoming UDP messages from a peer if no link is established.

Communicating over multicast UDP

The above paragraph (Communicating over UDP) remains valid for communication over multicast UDP.

No protocol is specified when choosing a multicast group. It is recommended to select a group at random.

Inspection and control of services

Service inspection is achieved by sending controlQuery events and receiving controlAnswer events over a control link to a service. The TCP port for the control link to a service can be determined using the service discovery mechanism.

Inspection provides information on service parameters, inputs and outputs. Control allows to modify the values of a service's parameters.

Services not implementing the control protocol must emit the following message immediately after any connection has been established with their control port:

<controlError id="00000000" type="not-implemented"/>

Communication over the control channel obey the companion XML Schema in bip-control.xsd.

For instance, the following query and answer show how to inspect a service parameter:

<controlQuery id="DADADEAD">
        <variable name="stars"/>
</controlQuery>

<controlAnswer id="DADADEAD">
        <variable name="stars">
                <value>123</value>
                <default>100</default>
                <type>integer</type>
                <access>read</access>
                <description>current number of
                   stars in the sky</description>
        </variable>
</controlAnswer>

The control protocol is still subject to change and is not normative.