Applications in the fields of virtual and augmented reality as well as image-guided medical applications make use of a wide variety of hardware devices. Existing frameworks for interconnecting low-level devices and high-level application programs include high-frequency event streaming frameworks as well as dynamically typed message passing systems. Neither of these approaches exploits the full potential for processing events coming from arbitrary sources and neither is easily generalizable. In

more »

... his paper, we will introduce a new multi-modal event processing methodology that uses dynamically-typed event attributes for event passing between multiple devices and systems. The existing OpenTracker framework was modified to incorporate a highly flexible and extensible event model, which can store data that is dynamically created and arbitrarily typed at runtime. The software architecture that is introduced in this work provides multi-modal event streaming, network and file logging support, and the ability to merge separate event streams. Due to the implementation of a self-descriptive event serialization mechanism, the architecture does not put any constraints on new data types to be used within the library. The main factors impacting the library's latency and throughput were determined experimentally and the overall performance was shown to be sufficient for most typical applications. Several sample applications were developed to take advantage of the new dynamic event model that is provided by the library, thereby demonstrating its flexibility and expressive power. In order to integrate such systems and devices in a flexible and extensible manner, it is necessary to have a software framework that provides a high-level abstraction of hardware devices and their associated low-level software drivers. This framework needs to efficiently handle streams of data in real time, potentially using a pipes-and-filters architecture. In addition, transparent network support is desirable for distributed systems. Software systems have been developed to give formerly independent devices the ability to cooperate in a straightforward way [2, 3, 6 -9, 11 -13, 15, 16]. However, many of these solutions do not exploit the full potential for processing events coming from arbitrary sources and are not easily generalizable. Prior work in this area can be divided into two different classes of architectures. The first class uses an efficient fixed data structure for the streamed events that can be accessed, copied, and processed with minimum delay. This is therefore suitable for high-frequency, real-time streaming of event data; however, the * Please send all correspondence to Jochen von Spiczak at jochen.v.s@gmx.net. predefined, fixed event structure is limited in its applicability (e.g., it may be designed exclusively for tracking, image or video streaming). A second class of architecture implements processing of multi-modal events, the content of which can be flexibly defined. The event format encapsulates data of different types in a potentially self-descriptive way. Such systems are often used for asynchronous event processing and typically achieve lower performance due to the associated overheads. Therefore, they can often more appropriately be described as message passing, rather than stream processing systems. In this paper, we introduce the concept of streaming multi-modal events with generic types. Following this approach, events are generated dynamically at runtime-as required by the application-and can contain data of arbitrary types, including composite high-level types defined by the user. The structure of each event can change during the lifetime of the event-information can be added, accessed, modified, and deleted. Event streams containing data from different data sources, which logically belong together, can be merged and handled as a single event stream. The approach is suitable for high-performance VR applications that require both high-bandwidth event streaming and support for multiple heterogeneous devices. This software framework has been released under the BSD Open Source license [1] and is freely available.