Gloria Elena Jaramillo¹, Juan Esteban Quiroz², Cesar Augusto Cartagena², Carlos Andrés Vivares², John Willian Branch³

¹ Ingeniera de Sistemas e Informática, Magíster en Sistemas e investigadora, Grupo de Investigación y Desarrollo en Inteligencia Artificial, Universidad Nacional de Colombia Sede Medellín. Medellín, Colombia. gejarami@unal.edu.co

² Ingeniero de Sistemas e Informática. Universidad Nacional de Colombia, Sede Medellín. Medellín, Colombia. jequiroz@unal.edu.co, cacartag@unal.edu.co, cavivaresm@unal.edu.co

Artículo recibido 27-IX-2010. Aprobado 8-XII-2010

Discusión abierta hasta junio de 2011

With the increasing power and miniaturization of mobile devices, emerging areas of study have found in such devices a new form of visualization and interaction. Augmented reality, as a promising metaphor of human-computer interaction, is not the exception. Although mobiles do not offer a feasible solution for several general purpose augmented reality applications, due mainly to hardware and processing limitations, some other applications have successfully used them for displaying virtual graphical information within real surroundings. In this paper, experiences gained with the use of augmented reality in mobile devices as a metaphor of visualization are presented. Specifically, we focus on the need to preserve cultural heritage and propose augmented reality as an interface for visualizing such cultural material. This paper offers two major contributions: the first one related to the fact of visualizing digitalized heritage of Colombian culture by means of an augmented museum and the second one, an enhancement of the user’s experience by displaying detailed reconstructed Colombian cultural pieces. We created an augmented museum as a case study and evaluated the viability of using handheld devices for visualization.

Con el poder creciente y la miniaturización de los dispositivos móviles, nuevas áreas de estudio han encontrado en esta tecnología una nueva forma de visualización e interacción. La realidad aumentada, como una prometedora metáfora de interacción humano-máquina, no es la excepción. Aunque debido ante todo a limitaciones de procesamiento y hardware, los móviles no ofrecen una solución viable para algunas aplicaciones de realidad aumentada de propósito general, otras aplicaciones los han utilizado con éxito para visualizar información gráfica virtual en entornos reales. En este artículo presentamos experiencias adquiridas con el uso de realidad aumentada en dispositivos móviles como metáfora de la visualización. Específicamente, nos concentramos en la necesidad de preservar el patrimonio cultural para proponer la realidad aumentada como una interfaz para visualizar dicho patrimonio. Este artículo presenta dos aportes principales: el primero, la forma de visualización de patrimonio cultural colombiano y el segundo, una mejora de la experiencia del usuario al presentar piezas culturales colombianas detalladas. Se creó un museo aumentado como caso de estudio y se evalúa la viabilidad del uso de dispositivos de mano para la visualización.

Augmented reality (AR) is defined as a technique that allows interacting with and visualizing virtual graphics on the top of the user’s view. Figure 1 shows the Milgram’s continuum (Milgram and Kishino, 1994), in which AR appears as a mixed reality environment, with one part being real and the other one virtual, where real environment predominates.

The specific characteristics of augmented environments make that this metaphor of visualization presents a lot of possibilities for human-computer interaction and for solving visualization and access information problems. Since its beginnings, several areas of study have used AR as a tool for solving these problems, generating promising applications. For naming only some examples, medicine has benefited of AR by reducing the need for invasive treatments or procedures. A typical practice of AR in this field consists in superimpose magnetic resonance imaging (MRI) or computerized tomography (CT) into the patient’s body. Figure 2a shows the reconstructed model of a skull superimposed into a real one (Mellor, 1995).

Due to the fact that AR could be used as a tool for solving information-access problems, AR presents a really clear application in annotation and visualization. By means of AR, the user can access to hidden textual or graphical information, seeing, for example, an occluded building or to obtain additional information about it. In tourism and military, the annotation of unfamiliar places is an important issue, and augmented reality offers the possibility of viewing such information by using video or optical see-through head-mounted displays (HMD), or a mobile device. Another application, close to the earlier, relies on manufacturing and repair. Annotations superimposing in the real user’s surrounding could be instructions used as visual graphical or textual manuals, making the understanding of the operation of a machine easier. One of the most cited works in these fields is Höllerer and Feiner’s work (Feiner, MacIntyre and Seligmann, 1993; Feiner, MacIntyre and Höllerer, 1997; Höllerer et al., 1999). Figure 2b shows a laser printer maintenance application derived from their work.

Applications such as cultural heritage, entertainment, robot path planning, urban modeling, journalism, and education are also common in augmented reality.

Additionally, the decrease of the size and increase of the power and processing in mobile devices has allowed the implementation of augmented reality applications in cell phones, PDA, iPad and other mobile devices. In this aspect, we specify the term mobile augmented reality. There are two possibilities, non exclusive, in which this term can be applied. The first one related to applications in outdoors that require mobility of the user. This mobile augmented reality often consists on wearable computers and special displays for the visualization (video or optical HMD). In this case, the system should be extremely robust to tolerate outdoor conditions: lighting changes, occlusions, and environmental conditions. The second definition is related to a system that can or cannot work in structured environments and use handheld devices for the visualization and interaction. The second one is the definition that we used for our case study.

Whatever definition used, working on mobile applications implies take into account some requirements. In this part, we follow the requirements presented in Höllerer and Feiner (2004):

• Display to present the augmented scenario.

• Tracking sensors to obtain information (position and orientation) of the user’s view. This can be achieved by means of inertial, magnetic, ultrasonic, or optical sensor (including traditional cameras).

• Wearable input and interaction technologies that enable to make selections or access to databases. Using wearable input is not an obligatory element of an AR’s architecture.

Currently, these requirements make the creation of a mobile augmented reality system a challenging task.

Taking it into account and considering the potential applications of AR as an interface that directly impacts human-computer interaction (HCI), we resume previous studies in our research group and propose as a case study the creation of an augmented museum that not only meets the need for preservation of cultural artifacts, but also allows high-quality viewing using all the techniques and technologies offered by the AR. In the present paper, we report the experiences gained using mobile augmented reality as a metaphor for visualization and use of an augmented museum as a case study. For that purpose, we present in section 2 some relevant works carried out in this area. In section 3, we will present our augmented museum and the methodology used. The results are shown in section 4, where we conclude.

As we stated in the previous section, due to the applicability of mobile augmented reality, a lot of works have existed that use it as a tool or as a study area by itself. Among those works, we found applications as SitePack (Nielsen, Kramp and Grønbæk, 2004), for visualization of architectural models. In this work, the mobile AR system is composed of a tablet PC with a web cam (for visualizing and processing of the visual information) and a GPS (for tracking requirements). This tool, thought for outdoors, allows dynamic creation of virtual objects while assessing the visual impact of a 3D model in architectural applications.

In Wagner and Schmalstieg (2003), ARToolkit is implemented in PDA in a Windows CE platform as a guiding system for buildings. The authors of this article state as main limitations the lack of precision in the generation of graphics, which is attributed to the absence of support to floating data structures in mobile devices. To overcome this non-trivial limitation, in the work proposed by Wagner et al. (2005), Klimt library is proposed. It uses a combination of OpenGL ES and WGL.

An alternative solution for overcoming processing limitation on handheld devices is the use of client-server architecture. In Pasman and Woodward (2003), a networking communication is used for sending a threshold image of the live video to the server through WLAN or GMS. ARToolkit is used for tracking and rendering procedures, and the processed information is sent back to the user. In Mooser et al. (2007), TriCode markers, fiducials composed of eight triangles forming 24-bit codes, are proposed and implemented in a Sony Vaio UX Micro PC. It has presented good visualization results. In both approaches, users face the problems of the delay in the data transmission and a possible data loss due to connectivity problems.

In Colombia, although using cutting-edge mobiles is very common, AR is yet a new research area. In museums, for example, the use of AR for the interaction and visualization is really new. For that reason, in this paper, we evaluate the impact of such a new metaphor in that specific application.

This research is divided into three stages: the first one related to the process of creating the virtual version of the cultural pieces, the second one related to the visualization of the pieces in an enhanced or augmented environment, and finally the implementation of the system in the handheld device.

3.1 Creation of the virtual cultural objects

One key aspect in the creation of a visually realistic augmented environment lies on the virtual models. In order to work with realistic models we do not directly create the virtual cultural models using any graphical pipeline. On the other hand, we follow a strategy based on tridimensional reconstruction of the real pieces in order to obtain more fine detailed and realistic models. This stage is subdivided in several stages including data acquisition from the real pieces, registration, adjustment, integration, and segmentation. These stages were developed following the methods and techniques described in Branch (2007).

Bearing in mind that the physical interaction with cultural heritage objects is, in most of the cases, impossible due to the material and natural deterioration, we pretend to avoid direct handling of the piece by getting information of them using non-contact sensors, particularly lasers. After data acquisition carried on the scanning stage using a laser, we obtained a set of range images partially overlapped, which are registered for obtaining only one tridimensional view of the object. For doing that, we implemented a genetic algorithm that matches the range views. Because of the morphology of the real object, it is possible to make errors in the data acquisition stage which are reflected in holes and redundant data. In the integration stage those errors are corrected. Finally, a mathematical model is adjusted to the model data attained from the integration.

The pieces used were reconstructed following the method mentioned, and fine detailed was obtained implementing the contributions in the reconstruction process reported in Branch (2007), namely: the implementation of a genetic algorithm in the registration stage, a robust method for automatically filling holes in the integration stage, and squaring of the triangular meshes in the adjustment. Figure 3 shows some of the reconstructed pieces with the mentioned process.

3.2 Augmented reality environment

The common process of creating an augmented environment using artificial marks consists firstly in identifying some aspects of the mark, in our case lines forming squares or rectangles and from them extract features that can be used in the tracking process. A frequent limitation reported in the literature when using artificial marks is that if part of the mark is occluded the system fails. In our particular application, we avoid possible occlusions, eliminating the need to move the mark, by carefully placing the marks and allowing the user to move freely in the scene and around the mark.

After the process of identifying the squares or rectangles, the intersections of those lines (vertices) become the input of the tracking stage. Tracking is an important part of the system because of what we mentioned, the user can move freely and that relative movement implies that the 3D coordinates of the mark relative to the user are changing in real time, so it is necessary to know during the augmentation process the coordinates in which the features (vertex of the square) are located.

Up this point, all the process has been completely geometric and no virtual object has been associated. Once we have estimated the coordinates in which the real object (mark) is located, the image in the mark is compared with the images pre-loaded in the system. The virtual object associated to that image is rotated according to the coordinates estimated, in order to appear coherently in the mark position, by means of a simple matrix operation using graphical pipelines and finally it is rendered. The described process is summarized in figure 4.

3.3 Handheld augmented reality system

The last part of this proposed augmented museum consists in a greater degree of interaction by embedding the system in a mobile device. This stage is not simply to embed the system in a new platform, since several aspects should be considered such as interface ports, multimedia capabilities, power consumption, technical support, price, and mainly graphical support and an operating system. It is for those aspects that this stage cannot be isolated from the other ones, and all aspects considered to create the previously described system were orientated to perform in a mobile platform. We use an Android device in this part because of its support to render graphics from OpenGL pipeline, and its robust and integrated operating system that allows the creation of mobile applications. At this point, a limitation we found was the size of the display. Although the system also runs in a conventional PC, we do not use it as a visualization display since movements could be restricted, and for some users, such as kids, portable computers may be uncomfortable and not ergonomic. Figure 5 shows the architecture implemented in this paper.

Because of the nature of this application, quantitative results about the visual perception are difficult to present. Because of this is why we implemented methods reported in the literature in order to evaluate the impact of this augmented museum in users (Baber et al. 2001; Damala et al., 2008). In general, surveys are a suitable strategy to demonstrate the hypothesis about the use of AR as interface of interaction. Quantitative evaluation of the performance consisted of surveys that asked about aspects such as efficiency, acceptation to the technology, ergonomics, and usability.

Results of the final handheld augmented reality system are shown in figure 6.

In this paper we presented experiences gained from the implementation of a mobile augmented reality system using as a case study an augmented museum. Although in Colombia approaches such as high-quality images and videos alongside the use of haptic interfaces have been proposed for the visualization of reconstructed 3D pieces, no work has been reported that used augmented reality for the visualization of cultural artifacts. It opens new possibilities of research not only in visualization but also in human-computer interaction. In addition, as it was established in the section 3.3 this technology has good acceptance by the users and becomes an attractive technology for them.

Being aware of the impact and usefulness of augmented environments, in this paper we present a prototype of an augmented reality system that seeks as an objective the popularization of this technology not only as an objective itself, but as a tool for solving problems of visualization or interaction in daily environments. This work is a contribution to the research on new human-computer interaction techniques that presents good results, but there is yet much work to do. Further research, in which we are working on, should be done in techniques that eliminate the need of using artificial marks in the scene.

Currently, the authors of this article are working on natural features in regions extracted from the eigenvalues of the frame and the user selects manually the position in which the virtual object should appear. In this proposed approach, 3D coordinates are estimated from affine projection as proposed in Pang et al. (2006). Another topic which we are working on is about the optimization in the rendering and possible handling of 3D complex models in limited resources, specifically, handheld hardware.

Baber, Chris; Bristow, Huw; Cheng, Sean-lee, Hedley, Anna; Kuriyama, Yuri; Lien, Marc; Pollard, James and Sorrell, Phil (2001). "Augmenting museums and art galleries". 13rd Conference on Human-Computer Interaction Interact 2001. pp. 439-447. Tokyo, Japan (9-13 July).        [ Links ]

Bederson, Benjamin B (1995). "Audio augmented reality: a prototype automated tour guide" ACM Human Computer in Computing Systems Conference, pp. 210-211. Denver, CO (7-11 May).        [ Links ]

Branch, John Willian (2007). "Reconstruction of free form objects from range images using a net of NURBS patches". PhD Thesis. Universidad Nacional de Colombia Sede Medellín.        [ Links ]

Damala, Areti; Cubaud, Pierre; Bationo, Anne; Houlier, Pascal and Marchal, Isabelle (2008). "Bridging the gap between the digital and the physical: design and evaluation of a mobile augmented reality guide for the museum visit". Proceedings of the 3rd International Conference on Digital Interactive Media in Entertainment and Arts. pp.120-127. Athens, Greece (10-12 September).        [ Links ]

Damala, Areti; Marchal, Isabelle and Houlier, Pascal (2006). "Merging augmented reality based features in mobile multimedia museum guides" XXI International CIPA Symposium, Athens, Greece.        [ Links ]

Feiner, Steven; Macintyre Blair and Höllerer Tobias (1997). "A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment". Proceedings of the First International Symposium on Wearable Computers, ISWC´97, pp. 74-81. Cambridge, MA (13-14 October).        [ Links ]

Feiner, Steven; MacIntyre, Blair and Seligmann, Dorée (1993). "Knowledge-based augmented reality". Communications of the ACM, vol. 36, No. 7 (July), pp. 53-62.        [ Links ]

Höllerer, Tobias and Feiner, Steven (2004). Mobile augmented reality. Telegeoinformatics: location-based computing and services. In: Karimi, H. and Hammad, A. (eds). Haarlem, The Netherlands: Taylor & Francis Books. Chapter 8, pp. 221-256.        [ Links ]

Höllerer, Tobias; Feiner, Steven; Terauchi, Tachio; Rashid, Gus and Hallaway, Drexel (1999). "Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system". Computers and Graphics, vol. 23, No. 6, pp. 779-785.        [ Links ]

Mellor, John Phillip (1995). "Enhanced reality visualization in a surgical environment". MIT, Cambridge. Technique Report 1544.        [ Links ]

Milgram, Paul and Kishino, Fumio. (1994). "A taxonomy of mixed reality visual displays". IEICE Transactions on Information Systems, vol. E77-D, No. 12 (December), pp. 1321-1329.        [ Links ]

Mooser, Jonathan; Wang, Lu; You, Suya and Neumann, Ulrich (2007). "An augmented reality interface for mobile information retrieval". Proceedings of International Conference on Multimedia and Expo. (ICME´07), pp. 2226-2229. Beijing, China (2-5 July).        [ Links ]

Nielsen, Michael Bang; Kramp, Gunnar and Grønbæk, Kaj (2004). "Mobile augmented reality support for architects based on feature tracking techniques". Proceedings of the Workshop on Interactive Visualization and Interaction Technologies, Krakow, Poland.        [ Links ]

Pang, Ying Han Han; Yuan, May; Nee, Andrew; Ong, Soh Khim and Youcef-Toumi, Kamal (2006). "A markerless registration method for augmented reality based on affine properties". Conferences in Research and Practice in Information Technology (CRPIT). 7th Australasian User Interface Conference, vol. 50. Hobart, Australia.        [ Links ]

Papageorgiou, Dimitra; Ioannidis, Nikos; Christou, Ioannis; Papathomas, Michael and Diorinos, Markellos (2000). "Archeoguide: An augmented reality based system for personalized tours in cultural heritage sites". Cultivate Interactive, issue 1 (3 July).        [ Links ]

Pasman, Wouter and Woodward, Charles (2003). "Implementation of an augmented reality system on a PDA". Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR´03), pp. 276-277. Tokyo, Japan (7-10 October).        [ Links ]

Renevier, Philippe and Nigay, Laurence (2001). "Mobile collaborative augmented reality: the augmented stroll". Lecture Notes in Computer Science, vol. 2254. Proceedings of the 8th IFIP International Conference on Engineering for Human-Computer Interaction, pp. 299- 316. Toronto, Canada (11-13 May).        [ Links ]

Thum, Marco; Demiris, Thanos and Müller, Stefan (2006). "A multimodal user interface component for an augmented reality mobile user guidance system". Proceedings of the 14th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision. Plzen, Czech Republic (30 January - 3 February).        [ Links ]

Wagner, Daniel and Schmalstieg, Ditier. (2003). "ARToolkit on the PocketPC platform". IEEE International Augmented Reality Toolkit Workshop, pp. 14-15. Tokyo, Japan (October).        [ Links ]

Wagner, Daniel; Pintaric, Tomas; Ledermann, Florian and Schmalstieg, Dieter (2005). "Towards massively multi-user augmented reality on handheld devices". Third International Conference on Pervasive Computing. Munich (8-13 May).        [ Links ]

White, Martin; Mourkoussis, Nicholaos; Darcy, Joe; Petridis, Panos; Liarokapis, Fotis; Lister, Paul; Walczak, Krzysztof; Wojciechowski, Rafal; Cellary, Wojciech; Chmielewski, Jacek; Stawniak, Miroslaw; Wiza, Wojciech; Patel, Manjula; Stevenson, James; Manley, John; Giorgini, Fabrizio; Sayd, Patrick and Gaspard, Francois (2004). "ARCO: an architecture for digitalization, management and presentation of virtual exhibitions". Proceedings of the Computer Graphics International (CGI´04), pp. 622- 625. Crete, Greece (16-19 June).        [ Links ] ]]> 2001 2001 439-447 1995 210-211 2007 2008 120-127 2006 Athens 1997 74-81 1993 36 7 7 53-62 2004 221-256 1999 23 6 6 779-785 1995 1994 E77-D 12 12 1321-1329 2007 ICME´07 2226-2229 2004 2006 50 2000 1 1 2003 276-277 2001 2254 299- 316 2006 2003 14-15 2005 2004 622- 625