language: en

[Proc. UIST '93 (ACM Symp. on User Interface Software and Technology), Atlanta GA, November 3-5, 1993, 145-155]

Windows on the World: 2D Windows for 3D Augmented Reality

Steven Feiner Blair MacIntyre Marcus Haupt Eliot Solomon
Department of Computer Science Columbia University
New York, NY 10027

[212-939-7000]

{feiner, bm, haupt, esolomon}@cs.columbia.edu

ABSTRACT INTRODUCTION We describe the design and implementation of a prototype When we think of the use of head-mounted displays and 3D heads-up window system intended for use in a 3D environ- interaction devices to present virtual worlds, it is often in ment. Our system includes a see-through head-mounted terms of environments populated solely by 3D objects. display that runs a full X server whose image is overlaid on There are many situations, however, in which 2D text and the user's view of the physical world. The user's head is graphics of the sort supported by current window systems tracked so that the display indexes into a large X bitmap, can be useful components of these environments. This is effectively placing the user inside a display space that is especially true in the case of the many applications that run mapped onto part of a surrounding virtual sphere. By under an industry standard window system such as X [13]. tracking the user's body, and interpreting head motion rela- While we might imagine porting or enhancing a significant tive to it, we create a portable information surround that X application to take advantage of the 3D capabilities of a envelopes the user as they move about. virtual world, the effort and cost may not be worth the return, especially if the application is inherently 2D. We support three kinds of windows implemented on top of Therefore, we have been exploring how we can incorporate the X server: windows fixed to the head-mounted display, an existing 2D window system within a 3D virtual world. windows fixed to the information surround, and windows fixed to locations and objects in the 3D world. Objects can We are building an experimental system that supports a full also be tracked, allowing windows to move with them. To X11 server on a see-through head-mounted display. Our demonstrate the utility of this model, we describe a small display overlays a selected portion of the X bitmap on the hypermedia system that allows links to be made between user's view of the world, creating an X-based augmented windows and windows to be attached to objects. Thus, our reality. Depending on the situation and application, the hypermedia system can forge links between any combina- user may wish to treat a window as a stand-alone entity or tion of physical objects and virtual windows. to take advantage of the potential relationships that can be made between it and the visible physical world. To make this possible, we have developed facilities that allow X KEYWORDS: augmented reality, virtual reality, virtual windows to be situated in a variety of ways relative to the worlds, head-mounted displays, portable computers, mobile user and the 3D world. computing, window systems, X11, hypertext/hypermedia. In this paper we first present related work and provide an overview of our system. Next, we describe the different kinds of windows that we support, and show how these windows can be used to advantage by a simple hypermedia system. Finally, we explain the underlying system architec-

[Permission to copy without fee all or part of this material is granted ture and describe our current implementation. provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. 1993 ACM 0-89791-628-X/93/0011 ... $1.50 November 3-5, 1993 UIST '93 145]