Inglês (pdf)
Artigo em XML
Referências do artigo
Tradução automática
Enviar este artigo por email
Citado por SciELO 
Citado por Google 
Similares em
SciELO 
Similares em Google 
Permalink
I. INTRODUCTION
Augmented Reality (AR) is a technology that allows users to observe virtual elements such as 3D objects in a real-world context. The virtual and real elements are part of the user interface (UI) and can be manipulated in real time. The development of augmented reality applications has grown over time, becoming highly impactful in domains such as engineering, medicine, and education [1-3]. Part of these applications are designed from scratch by using low-level toolkits, but when dealing with the creation of complex augmented reality systems, this approach becomes error-prone and more time-consuming to develop [4].
The user interface is one of the main components of an augmented reality system whose design occasionally becomes a challenge for the developer [5]. Its implementation is a time-consuming task when performed at a low programming level so tools that facilitate and speed up its development become necessary [6]. Oliveira and Araujo [5] mention two main approaches to developing user interfaces for augmented reality: frameworks and authoring tools. Frameworks try to cover the whole process of implementing AR applications although they focus on application logic instead of user interface.
Authoring tools offer a customized set of interface and interaction components to develop applications without needing programming skills. Some tools use high-level description languages to describe user interfaces in text files. Generally, the syntax of these languages is XML-based [7]. This paper aims to identify languages used to describe user interfaces in AR development. To achieve this, a systematic literature review was carried out.
This paper presents the review results of the languages used to describe augmented reality user interface. Also, new criteria were proposed to extend the framework presented by Córdova Gonzales [8], and a comparative evaluation of the languages is performed.
II. METHODOLOGY
The research aim is to find languages used to describe user interfaces in augmented reality applications, and, through a comparison, select the language that is best suited for the creation of three-dimensional interfaces for augmented reality. To find relevant papers, the method proposed by Kitchenham and Charters [9] was applied in the review. The authors suggest the review implementation in three phases: planning, conducting, and reporting.
A. Planning the Review
Firstly, a search in several databases was carried out by using associated keywords with search objectives. Once a set of articles is obtained, a filter based on the criteria was applied to identify the papers that can provide direct evidence to the review question: What languages have been developed to describe the user interfaces in augmented reality applications? The following databases were used in the search for papers: Springer, IEEE, ACM, and EBSCO Discovery Service. A set of search strings was used, as seen in table 1. The items used were title, abstract, keywords. Results by each database can be seen in Table 1.
Table 1 Results by each database according to search string.
B. Conducting the Review
Figure 1 shows the process carried out in the selection of papers. A total of 1,156 results were obtained, excluding duplicated papers. The Springer search was refined for each equation considering the subdiscipline "User interfaces and Human-Computer Interaction", excluding 408 results and limiting the total to 363.
A set of 74 papers was selected from the 748 by analyzing the relevance of the title and abstract. A new set of 14 papers found by online search engines (Google Scholar and Semantic Scholar) using the search strings was included and considered relevant to the research.
The following selection criteria were applied to a collection of 88 papers: papers that involve the use of languages in the user interfaces development; papers that present methods or tools that contribute to the development of augmented reality applications; papers not related to augmented reality. Following the criteria, 56 papers are excluded, and 8 papers are included due to snowballing, resulting in 40 papers total.
C. Reporting the Review
After analyzing the papers obtained in the review, 10 languages were identified, most of them to specify or store AR content. To have an overview, a synthesis of each language is presented, focused on features, limitations, and application examples.
Table 2 Languages identified in the review.
| Language | Authors |
|---|---|
| ARML 2.0 [11] | Open Geospatial Consortium |
| APRIL [12] | Florian Ledermann, Dieter Schmalstieg |
| CARL [13] | Dariusz Rumiński, Krzysztof Walczak |
| AREML [14] | Rafał Wojciechowski |
| SSIML/AR [15] | Arnd Vitzthum, Heinrich Hussmann |
| KARML [16] | Blair MacIntyre, Alex Hill, Hafez Rouzati, Maribeth Gandy, Brian Davidson |
| ICDL [17] | Pablo Figueroa, Raimund Dachselt, Irma Lindt |
| ARUIML [18] | Mikel Salazar Gonzalez, Carlos Laorden Gómez |
| MRIML [19] | Wolfgang Broll, Irma Lindt, Jan Ohlenburg, Axel Linder |
| CUIML [20] | Christian Sandor, Thomas Reicher |
1) Augmented Reality Markup Language (ARML). ARML [11] is an XML-based data standard for describing and interacting with AR scenes. ARML 1.0 was a development document that extended a section of KML (Keyhole Mark-up Language) [21] to improving augmentation in location-based AR services. The second version [22] is based on standards such as KML, GML (Geography Markup Language) [23], and COLLADA (Collaborative Design Activity) [24]. ARML 2.0 aims to describe an AR scene, the objects contained in the scene, and their behavior. It has three concepts: Features represent the physical object to be augmented, VisualAssets describe the virtual object appearance in the augmented scene, and Anchors describe the spatial relationship between the physical and virtual object. The current specification focuses on visual objects, and its implementation was limited to mobile devices [25]. Córdova Gonzales [8] presents the development of an augmented reality browser for vision-based applications using markers where each scene is described under the ARML 2.0 standard.
2) Augmented Reality Presentation and Interaction Language (APRIL). APRIL [12] is an XML-based language for creating augmented reality presentations within the Studiertube framework [26]. By using APRIL, authors can specify multimedia objects, behaviors, and possible user interaction. UML state diagrams are used to specify the flow of an augmented reality presentation; they are stored as XMI files and then translated into APRIL format. From this description, Studiertube configuration files are generated that allow running the augmented reality presentation on that platform. Ledermann and Schmalstieg [27] present several application examples, such as an interactive storybook for children, a virtual display system for archaeological artifacts, and an outdoor tourist guide application. APRIL is not based on task-centric domains [28] and some design aspects are also not considered [4, 15] -such as the 3D content structure and the way real and virtual objects are organized within the hierarchical structure of the user interface. Although Ledermann and Schmalstieg state that it should be possible to use other platforms to reproduce presentations created with APRIL [27], Vitzthum and Hussmann mention that the presentations are not intended to be platform-independent as they are designed to run on the Studiertube platform [28].
3) Contextual Augmented Reality Language (CARL). CARL [13] was designed to enable the modeling of Contextual Augmented Reality Environments (CARE) [29] that support both directional and natural augmentation. CARL allows the specification of content objects, trackables, and interfaces by using XML syntax. Unlike other languages such as ARML and APRIL that help simplify the development of augmented reality environments, CARL was intended for dynamic AR contextual information visualization systems. A case study [30] presents a CARL browser application for a bookstore. Ruminski [31] introduces the concept of spatial sound modeling implemented in the CARL browser allowing augmented reality presentations enriched with 3D sound.
4) Augmented Reality Environment Modeling Language (AREML). AREML [14] is an extension of MR-ISL (Mixed Reality Interaction Scenario Language) [32] which is a high-level language for defining interaction scenarios in augmented reality environments. AREML was developed following the AREM (Augmented Reality Environment Modeling) approach for creating AR environments inspired by the object-oriented paradigm. Its development is based on ARSM (Augmented Reality Scene Model) which formalizes the concepts of AR-Class, AR-Object, and its components through the AREML language.
5) SSIML extension for Augmented Reality (SSIML/AR). Scene Structure and Integration Modeling Language [33] is a language aimed at the integration of 3D scenes in complex applications. SSIML/AR [15] is an extension of SSIML adapted to structured specification and development of AR applications, focusing on AR user interfaces. An example application presented in the article shows a simplified scenario of an AR system that guides the user through installing a graphics card on a desktop computer [15].
6) KML AR Markup Language (KARML). KARML [16] is an extension of KML (Keyhole Markup Language) which is an XML language focused on geographic visualization. KARML is part of the KHARMA architecture [34] that focuses on reconceptualizing the KML language within the context of AR in browsers for mobile devices. Although the authors attempted to limit modifications to the existing standard to allow AR to use cases, they modified the semantics of KML as much as necessary and ended up moving away from the original KML specification. The authors of KARML joined the ARML 2.0 SWG to develop ARML [35].
7) Interface Component Description Language (ICDL). ICDL [17] is an XML-based specification language that enables the description of both generic and specific 3DICs (3D Interaction Components). It encapsulates a conceptual framework for 3DICs encompassing 3D widgets, 3D gadgets, and 3D interaction techniques. ICDL implements elements from other XML languages such as InTml [36] for status graphs, bibliographic references, and documentation. A ring menu defined in XML is shown as an example of the implementation of a specific 3DIC.
8) Augmented Reality-based User Interaction Markup Language (ARUIML). ARUIML [18] is an XML-based descriptive data format that facilitates the development of the presentation layer of AR applications with a user-centric approach. It provides communication mechanisms to support meaningful and engaging interactions with end users. ARUMIL is based on an interaction model composed of users, client systems, server systems, spatial layers, and non-spatial layers. For the adaptability of user interfaces to different situations, ARUIML proposes a structure of context-sensitive layers following a model-based approach.
9) Mixed Reality Interface Markup Language (MRIML). MRIML [19] is a language created to define cross-platform user interfaces, object behavior for interaction prototypes in virtual reality or augmented reality, and object-oriented scene diagrams for the development of complex application scenarios. MRIML provides a vocabulary for defining UI elements in a document that is sent to the rendering unit which create UIs for individual platforms. As an example, the article presents two rendering results (2D and 3D) of a user interface from the same sample scenario.
10) Cooperative User Interfaces Markup Language (CUIML). CUIML [20] is a high-level abstraction description language for views and controllers of the MVC (model-view-controller) design pattern that can be transformed through XSL transformations into required structures at runtime. Since CUIML is XML-based, it can be easily extended to support many different portable systems. This language was published in 2001 and is the oldest of the languages found in the systematic review.
D. Related Work
Visser [7] shows a study of XML languages used in augmented reality to store content, which sought the possibility of standardizing the format of AR content to allow interoperability between different AR browsers. Ahn [37] presents a content structure for AR application development in HTML5 for mobile devices, which seeks a separation between AR content and application logic. The article also shows a comparison between seven AR markup languages. The Open Geospatial Consortium (OGC) [35] introduced the concepts of ARML 2.0 and a comparison with other AR browser content formats. Córdova Gonzales [8] analyses the differences between ARML 2.0 and other augmented reality languages.
III. COMPARATIVE ASSESSMENT OF AUGMENTED REALITY LANGUAGES
Córdova Gonzales [8] proposes seven evaluation criteria to compare augmented reality languages, based on the studies conducted by Visser [7] and Ahn [37] who propose eight and five criteria, respectively. The criteria proposed by Visser focus on the requirements that an augmented reality language must meet to be a candidate for standardization, while in Ahn [37], they focus on the content structure of each language. The authors decided to use the seven criteria of the first work since they derive from the two previous works. The criterion "TESTAR - Test Definition" was discarded as the documentation is insufficient to evaluate it, which resulted in the addition of the criterion "DOC - Amount of Documentation". The criterion "FOCUS - Language Focus" was added since some of the languages found have a specification more focused on content description than on user interfaces. Finally, the criterion "YEAR - Year of publication" allows to organize them by date.
A. Description of Evaluation Criteria
ARTYPE - Augmented Reality Type: This criterion verifies the types of augmented reality supported by each of the languages. The following types are defined: G for geolocation-based augmented reality; M for marker-based augmented reality; N for augmented reality based on marker-less vision.
VRTUAL - Virtual Objects Type: It defines the types of virtual objects that are supported by each language.
STRUCT - Document structure: This criterion defines the base technology on which the document is based.
VISUAL - Control over visualization: It identifies the forms of language control in which augmented reality objects are visualized. Parameters such as orientation, position, scale, and others such as color and shape are included. The following types are defined: T for control by tags defined in the language; C for control by CSS.
SCRIPT - Scripting language component: It defines whether the language provides any specifications for script-based components to model the interactions of an augmented reality application.
STANDR - Standards: This allows to validate whether the proposed language is a standard or not.
FOCUS - Language focus: It estimates whether the language developed was more focused on the description of user interfaces or the description of augmented reality scenes: S for increased focus on the description of augmented reality scenes; UI for increased focus on the description of user interfaces.
DOC - Amount of documentation: It evaluates whether the documentation found for the language is extensive and sufficient.
: Full documentation of language specification, implementation details;
: Incomplete documentation, e.g., the vocabulary of the language is not fully defined;
: Insufficient/no documentation available.
YEAR - Year of release: The year in which the version of the language was published.
IV. RESULTS
ARTYPE: 50% of the languages support geolocation-based augmented reality. Likewise, 60% support AR based on markers, and 80% AR based on vision. Finally, only one language (ARML) supports all types of AR. STRUCT: 70% of the languages are XML-based, although KML and UIML are markup languages that are also XML-based. VISUAL: All languages have their own tags defined to control features provided by each language. Only 20% uses CSS control. SCRIPT: Only 40% of the languages make use of scripts for interaction modeling, the remaining percentage uses other types of mechanisms. FOCUS: 50% of the languages were developed with a greater focus on scene description, the remaining 50% were more noticeable in the description of user interfaces. STANDR: ARML is the internationally accepted standard for augmented reality, published in version 2.0. YEAR: 60% of the languages were published between 2001 and 2006, the remaining 40% between 2010 and 2015, with the most recent publication in 2015. DOC: 40% of the languages lack documentation. Another 40% have documentation that does not cover the entire language specification. Only 20% of the languages have complete documentation.
Table 3 Comparative table of AR languages.
| Language | ArType | Vrtual | Struct | Visual | SCRIPT | Focus | Standr | Year | Doc | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| G | M | N | T | C | S | UI | |||||||
| ARML 2.0 | x | x | x | HTML, Text, Image, 3D Objects | XML | x | x | x | x | x | 2015 |
|
|
| CARL | x | x | Image, 3D Objects | XML | x | x | 2014 |
|
|||||
| ARUIML | x | x | Text, Image, 3D Objects | XML | x | x | x | 2013 |
|
||||
| AREML | x | x | Image, Video, Sound, 3D Objects | XML | x | x | 2012 |
|
|||||
| KARML | x | x | x | Image 3D Objects | KML | x | x | x | x | 2010 |
|
||
| ICDL | x | Image, 3D Objects, Hyperlinks, BibTex entries | XML | x | x | 2006 |
|
||||||
| SSIML/AR | x | x | 3D Objects 2D Text Objects | UML | x | x | 2006 |
|
|||||
| MRIML | x | x | Text, Image 3D Objects | XML | x | x | x | 2005 |
|
||||
| APRIL | x | x | Text, Image Sound, Video | XML | x | x | 2004 |
|
|||||
| CUIML | - | - | - | HTML, Text, 2D graphics, 3D graphics | UIML | x | x | 2001 |
|
||||
V. DISCUSSION
A languages comparison was carried out based on the evaluation criteria. The evaluation criteria were widened considering the information in papers about each of the AR languages. However, the lack of language documentation was a factor that made difficult the criterion application: The information provided by CUIML language was insufficient to know the type of augmented reality supported. The ARUIML paper does not present examples of language application. Several languages are published in a single article which falls short of covering the entire language specification. In contrast, ARML is defined in the OGC® standard [11], and Ledermann [12] expands on the key concepts and features of APRIL. Although ARML and APRIL are the only languages with extensive documentation, they have platform limitations: ARML development was focused only on mobile devices, and APRIL presentations can only be used on the Studiertube platform. Several languages did not resort to the use of scripts to model interactions in an augmented reality environment.
For SSIML/AR, an extension called SSIML/Behaviour [38] allows a detailed description of the behavior and animation of graphical objects. In ICDL, a state machine describes the behavior of a 3DIC. In CARL, one of the three main elements that compose its specification is the interface element that contains all possible interactions between the AR scene and the end user. In APRIL presentations, there is a central behavioral scenario for the entire scene. However, following this approach could limit the number of possible complex interactions.
The publication date of the language is an important aspect. Although ARML had its first specification in 2009, its second version became approved by the OGC as an AR data standard in February 2015 [11]. The standardization of ARML may be a possible explanation for the absence of AR language publications after that date. On the other hand, Córdova Gonzales [8] states that "very few browsers and augmented reality platforms have made efforts to implement this standard (ARML), nor is there any news about future implementations". Since its approval, ARML 2.0 did not achieve the expected adoption or commercial impact that it was promising. However, Liao [39] states that this reaction is frequently for standards, especially those created by consortia that make biased assumptions about future user practice and markets, which in some cases may be outweighed by the rate of technological change at the time the standard is approved.
VI. CONCLUSIONS
Considering the current development of augmented reality and the increase in the development of products based on this technology, knowing the languages that allow describing the user interfaces for this technology is a necessity. Due to this, a systematic review of scientific literature was carried out, where the existence of ten (10) languages used to describe the user interface for augmented reality applications was established. Likewise, a detailed analysis of each language was performed, focusing on the main characteristics, limitations, and some examples of their application. A comparative evaluation was conducted to establish the robustness of the language in terms of current development requirements. For this purpose, a comparative evaluation framework was adapted by adding the necessary criteria for the adequate and updated use of the language. The main findings of this study show that the lack of documentation is one of the major limitations since it prevents the in-depth evaluation of some languages. In addition, it was identified that these languages slowed down their development in 2015, and the adoption of the ARML standard has not been as expected. Finally, ARML 2.0 and APRIL are the languages that best met the evaluation criteria.
