Pervasive Display Systems

FFG Call: Research Studios Austria

Summary: Ranging form sub inch, wrist worn watch displays, to double inch size displays embedded into mobile phones and PDAs, to 10"- 30" displays of traditional personal computer systems, 40"-100" displays in entertainment, advertising and education systems, up to huge, 500-1000m² public screen systems, displays are a truly pervasive technology surrounding us in almost any life situation.

The objective of the proposed Research Studio Pervasive Display Systems (DISPLAYS) is the development of a highly-innovative framework for the networked, decentralized operation of heterogeneous, long-lived and adaptive DISPLAY services. The framework will be grounded on a foundational re-thinking of the linkage among physical display systems and current display technologies (CRT, LCD, PDP, OLED, projection based, LCos or DLP) together with the associated infrastructures (IP networks, wireless communication, BT, UMTS, WiFi), and the presentation logic coming form display service models, content management systems, and online data repositories (WWW). Understanding networks of physical displays as "information ecosystems", constituted by situation-aware, cooperative ensembles of interaction entities among humans and their envi-ronment, DISPLAYS will (i) define an innovative model for display services, content components and userinteractions, (ii) studying and experimenting display ecosystem architectures to enforce various forms of spatial selforganization, self-composition, and self-management for display data and services, and (iii) implementing an innovative infrastructure for the deployment and execution of display services, and for the management, trading, delivery and displaying of content.

The effectiveness of the proposed solutions will be made available to top Austrian technology inno-vators within three distinct lines of research (RLs). In close cooperation with stakeholders in public advertising, prime advertisers, event organiser, culture institutions, content authorities and ICT solution providers, DISPLAYS will address the following F&E challenges:

  • The objective of RL1, Display Environment Sensing (DES), is the development of "situation aware" displays, i.e. employing hardware and software sensors together with a distrib-uted data fusion and context recognition system, so that decision about content to be displayed is based on the knowledge about the particular situation (number of viewers, loca-tion, time of day, weather, etc.).
  • RL2, Interactive Displays (ID), strives for explicit interaction modalities among viewers and displays situated in public places (airports, train-stations, shopping malls, sports arenas, event settings).
  • Finally RL3, Unconventional Displays (UD), will research on "screen-less" displays, i.e. dispersed patchworks of seamless and subtle embeddings of dynamic display fragments into the architectural shape of a building, so that the overall appearance of those display patches altogether gives a consistent and "closed picture".
The proposed research studio DISPLAYS will create novel display frameworks (hardware, software and tools) based on new scientific knowledge and the experience of the Institute for Pervasive Computing at the University of Linz, and its well developed alliances and partner companies. The resulting pool of solutions will significantly change business processes in the advertising and media industry, will accelerate access to novel, emerging advertising markets, and enable to gain a com-petitive advantage in established markets through innovation

Partners: EPAMEDIA, CocaCola Österreich, Enterprise Sports Promotion, Geodan Salzburg, OÖ. Theater und Orchester GmbH

Status: started December 1st, 2008

Research Competence

The Institut für Pervasive Computing (IPC) at the University of Linz, over its seven years of existence has addressed display related research with the application oriented frontiers of its networked embedded systems research activities. The individual lines of research have been and continue to be: (i) Display Ensembles, i.e. constituted by a set of networked embedded, cooperative display elements, (ii) See-Through Display, i.e. context aware overlaying realities with visual con-notations based on augmented reality technology, (iii) Mobile and Wearable Displays, i.e. display systems at an integration and miniaturization degree allowing for any-time any-place solutions, (iv) Peripheral Displays, i.e. subtle and unobtrusive displays delivering information to the periphery of attention of the user, (v) Non-visual Displays i.e. displays not addressing visual, but other modalities of perception like vibro-tactile displays or olfac-tory display, and (vi) User- Displays Interaction. The primary motivation behind these lines of research have always been to demonstrate display system architectures, middleware frameworks, and real-time content management with display technology seamlessly, in-visibly and unobtrusively embedded into the environment or the respective artefacts, or in other words, displays that are not being perceived as “computer screens” in the first place. Critical factors for the development of showcases in the past, therefore, have always been the detection and matching of innovation demands with new solutions. Emphasising a technology approach, and cooperative effort with the leading industries (Siemens AG, Telekom Austria, T-Mobile, ONE, Silhouette AG, etc.), several display system solutions and fully functional prototypes have evolved out of this engagement. To illustrate the span of research issues, the degree of innovation and industrial or market relevancy contained in the solutions, as well as the level of technicality achieved with proposed showcases, a few of them are presented in following list:

Everywhere Displays (Digital Graffiti): The Digital Graffiti software framework combines positioning, geo-referencing and interaction components on a mobile platform, and lets users co-notate places with digital information generated with image or text input systems (camera enabled mobile phone), or simply with hand gestures (a). "Graffitis" are retrieved by physically approaching their vicinity (b), can operate active services like an "unvisible" parkometer (c), or viewed from a distance (d). (2007, [NPFK07], patented, in cooperation with Siemens AG CE SE1, Ars Electronica, JKU Institut für Wirtschaftsinformatik)

Video See-Through Displays (INSTAR Navigation): The Augmented Reality In-Car Navigation System based on real-time augmented reality overlays of route information and camera live stream display background (a) introduces a whole new driver experience for car navigation systems: user attention while directed to the onboard navigation system displays stays connected with the road live scene (c) (d). Besides video see-through on the dash board built-in navigation screen, prototype projection systems onto the wind shield (b) have been implemented (optical see-through). (2004, [CFNP04], [NPFK04a] [NPFK04b] [NPFK04c], patentend, in cooperation with Siemens AG CE SE1, Ars Electronica, JKU Institut für Wirtschaftsinformatik)

Optical See-Through Displays: The co-notation of physical objects (a) with metadata and multi-media annotations is perceived via DigiScope, a 6DOF holographic, optical see through system (b), implementing an "Invisible World Window" (c), with "physical world" browsing capabilities (d). (2003, [Fers03a] [Fers03b], proposed as "Smart Tower", a window based air-traffic control panel, to FREQUENTIS)

Public Displays (WebWall): Aiming at a first, interactive, seamless access public display installation for a mobile-communications provider, the WebWall framework enables anonymous n:n communication via shared public displays (airports, trainstations, concerts, sports- and culture events, etc) ©. A modular component based SOA software frameworks implements ad-hoc communication ("Post-it" notes), buy/sell scenarios (auctions), multimedia memory (picture gallery, video), public opinion polls, public web browsing, etc. (b), while strictly separating the physical display technologies from the distributed/networked presentation logic (a), and supporting a variety of mobile / wearable access technologies (mobile phone, handwritten character recognition pen) (b). (2002, Best paper award at PERVASIVE 2002 [Fers02a] [Fers02b] [Fers02c], in cooperation with ONE, Connect Austria GmbH)

Multimedia Facades: Early work of IPC towards semantically meaningful (context aware, viewer controlled, interactive, remote controlled, lighting sensitive) architecture annotation has demonstrated screen-less multimedia facades. Examples are the "Theatre Gravity" framework (a) (b) (c) (e), which models arbitrary shapes of the appearance of a façade as the playground for visual phenomena adhering the laws of gravitational physics. A seamless embedding of real-time streaming media content into the structure and ornaments of buildings has been prototyped for the Landestheater Linz, as a preparation and show case for scenarios to be integrated into the façade of the new Linz Opera currently under construction.  (2008, in cooperation with Oberösterreichische Theater und Orchester GmbH, Linz)

Liquid Displays (WaterWall): IPC has studied soft surfaces like fog, vapour and crystal liquids as an alternative to screen hardware, particularly for display systems that inherently gain from the soft nature of liquids and the randomness of the dynamics of liquid and steam dispersion. In empirical studies, like the 200 m2 fountain screen in a water pond (a) (c), we assess the utility of controlled and uncontrolled material propagation (b) for the purpose of displays (d). (2005, [Fers06a] [Fers06b]] in cooperation with Lange Nacht der Forschung)

Peripheral Displays (Informative Art): Peripheral displays aim at providing users with information considered relevant at arbitrary points of work or living engagement, in an easy and quickly to convey, aesthetic and sometimes even artful style (a) (b) (c). Adhering principles of visual perception, visualization and design, non-critical information coming from various different hardware and software sensors is aggregated through abstraction and selective omission, and displayed at the periphery of a user's attention. IPC has developed a peripheral display software framework aiming at intuitive awareness (information can be perceived at quick, opportunistic glances), decreasing anxiety and stress (prevent form actively having to find information), operating nondistractingly (the user can remain focussed with his primary task), and reducing cognitive load (both the preattentive, as well as at the interpretative phase of perception). (2007, [Fers07a] [Fers07b], in cooperation with Siemens AG, CT SE2.)

Remote Team Awareness Displays: Based on the peripheral display software framework, a Distributed Work-Team Awareness Support systems (Aquareness) has been developed. Remote sensors in real-time collect awareness information from different company sites (Munich, Princeton, St. Petersburg and Beijing), and metaphorically (Underwaterworld) deliver a „keep-in-touch" awareness sentience among the dislocated teams, in a privacy protecting, aesthetically pleasing installation. The encoded concepts cover social awareness (information about presence and activity of work team members), remote team awareness (agenda team members are engaged in, degree of busyness and readiness for interaction), cultural and environmental awareness (e.g. time zone, time-of-day, weather) and remote context awareness (lighting, noise, activity), the interaction is based on tangible user interfaces (fruits, sun flower), and the architecture is based on the Peer-it [Fers08a] [Fers07e] platform. (2008, in cooperation with Siemens CT SE2)

Smart Signs: Implicit (a) (b) and explicit (c) (d) interaction principles have been studied in a role and profile based, autonomic, P2P software framework. Scenarios within the "Digital Aura" framework implement action free spontaneous interaction based on BT wireless communication (implicit event/announcement calendaring from "Smart Posters" (a), personalized ads from "Smart Shop Windows" (b), or multimedia download from "Smart Movie Posters"), or physical hyper-referencing using visual tags (d). (2004, [Fers04], in cooperation with Siemens CT SE2, ONE, Connect Austria GmbH)

Physical Interface Controls: Tangible user interfaces (TUIs) give physical form to digital information, employing physical artefacts both as representations and controls for computational media. IPC has developed TUIs as a means for explicit interaction with systems that unify input (state machine based on hand gestures recognized from embedded accelerometers and gyroscopes) and output (displays) into a single artefact (b).  ((2004, 2008, [Fers06c] [Fers07d], patent submitted, in cooperation with Siemens CT SE2)

Mobile Displays (SMS Notification Services): While with UoVienna, Ferscha has developed the (worldwide) first RFID based sports timing notification system based on SMS, first presented at the Vienna City Marathon, among Europe's largest annual sports events, in the year 2000. 128 KHz transponder technology is used for implicit timing of athletes, and notifying race results in real-time over worldwide GSM networks (about 120K-150K SMS events handled within a few hours of race). The solution is unique in concept and performance, and is now operated by IPC for the biggest marathon events in Europe. (1999-2008, in cooperation with Vienna City Marathon, Berlinmarathon, Hamburgmarathon, Hannovermarathon, etc., T-mobile, SwissCom, ONE,  telering)

Vibro Tactile Displays: Among the primary human senses, vision, audition and tactation, particularly the sense of touch appears underemployed in todays designs of displays. While about more than 78% of the information perceived by humans is delivered via the sight and hearing channel, only about 21% is perceived via the haptic sense. IPC investigates, for situations of work or engaged activity, where both the visual and auditory channel are occupied because of the involvement in the foreground task, how notifications or alerts coming from the background can be delivered via the haptic channel, particularly for situations where important notifications tend to be "overseen" or "overheard". Prototypes range from body worn (vibra belt (a), vibra helmet (b)), to chair based solutions (c) (d) (home, office, car), and are built with microelectronics controlling embedded tactors (e) (f). (2008, patent submitted, in cooperation with Siemens CT SE 2)

Wearable Displays (SPECTACLES): A layered, framework oriented development approach is followed with the wearable displays platform SPECTACLES: the mechanics/optics layer involves the development of optical see through solutions; the electrical/digital layer is concerned with the physical characteristics of the computing modules (e.g. ARM PXA 271), bus characteristics, power dissipation; the software and OS layer (embedded Linux) hosts device drivers and FPGA programs, defines  memory layout, provides services like the system daemons, widget libraries, and utilities; the framework service layer contains all service components (media management) offered to the application layer. The SPECTACLES (hardware) modules like see-through optics, camera and video unit, text2speech unit, biosensor unit (HF, ECG, EMG), voice recognition unit, head gesture unit, wireless communication unit (BT, Zigbee, GSM, UMTS, WiFi), global positioning unit (Sirf III), acceleration and motion unit, etc. allow for purposeful reconfiguration of the appliance (sports, medical care, emergency, tourism, entertainment). (2005, 2008, patent pending, In cooperation with Silhouette AG)

Tangible Display Controls (TA-Cube): The TA-cube project as a radical, from-scratch design of a universal remote control  based on TUI principles illustrates the various involved competencies of IPC, ranging for hardware design (a) and electrical engineering (b), software development and microprogramming, interaction design (c), and industrial design and deployment engineering (d).  The cube is a ready to market, universal remote control built on ATMega168 micro-controller based electronics involving a three axis acceleration sensor and a gyroscope, together with low power IEEE 802.15.4 wireless communication components, in an ergonomic, aesthetically pleasing design. (2007, [Fers08b], in cooperation with Telekom Austria)

Direct Manipulation Displays: The direct manipulation of switching and control abstractions within a display is a potentially effective means to configure home and living environments, like selection (a), placing and grouping (b), intensity (c), or orientation adjustment (d). IPC has developed an OSGi framework supporting the configuration of home automation systems via UPnP in "Smart Living" installations. (2008, patent submitted, in cooperation with Siemens CT SE2)

Funding: FFG Call: Research Studios Austria

Duration: 2008 - 2011

Goal-Oriented Opportunistic Sensor Clouds
The 2nd International Symposium on Secure Virtual Infrastructures (DOA-SVI'12), 10-14 Sept 2012, Roma, Italy, Springer Verlag, September 2012.
Real-Time Gaze Tracking for Public Displays
First International Joint Conference on Ambient Intelligence (AmI-10), Malaga, Spain, Springer LNCS, Berlin-Heidelberg, 10 pages, November 2010.
Tactor Placement in Wrist Worn Wearables
14th annual IEEE International Symposium on Wearable Computers (ISWC'10), October 10-13, Seoul, South Korea, IEEE Computer Society Press, 8 pages, October 2010.