I. Introduction

Open data helps government institutions disseminate information of interest to civil society in order to provide transparency and social control, and thus, empower citizens through information access, to the point that today this philosophy of openness has transcended to other areas such as academia and research institutes, who seek the development and improvement of services, plans, programs, projects and standards with the collaborative participation between state-citizen-company.

The “open” data must have technical and legal characteristics to be used, reused and redistributed by any person or entity, without any restriction; These parameters are stipulated in the International Open Data Charter [¹].

In favor of this initiative, in some countries standards and portals have been implemented in order to contribute to its use; For example, in Colombia, Law 1712 of 2014 obliges all public entities to disclose their data, and since 2016, the nation adopted the principles established in the International Open Data Charter, making the Colombian State Data Portal available as a space for the dissemination of public information in the country [²]. Likewise, portals were created at the departmental and municipal levels, with the objective that each entity had its own space for data opening. In 2019, a total of 30 portals focused on the dissemination and access of open data were registered.

However, having quality open data portals implies that they fulfill a dynamic role in the data life cycle and that they establish a relationship between producers, publishers and data consumers, through interaction mechanisms that contribute to aspects such as identification of the demand for data, data publication of interest for specific users, the feedback of data sets and the portal, as well as the improvement of their quality.

At international level there are several proposals from experts in the area for the evaluation of open data portals, each with different dimensions, factors or aspects to carry out this process. Therefore, and given that the portal is the means by which the quality of the published data is guaranteed, the search is facilitated by the users, the data is available in usable formats and these are published so that respond to a specific demand in order to meet specific needs that generate value, for which they have an integral evaluation methodology with criteria and dimensions proposed by experts in preliminary work.

II. Portals Evaluation Methodologies

In order to have the necessary basis for the formulation of an evaluation proposal that covers the different perspectives, as well as to create a wider and complete evaluation mechanism, a documentary review of works and research was developed in the recent years for portals evaluation.

In most of the works, an implementation of the Tim Berners-Lee five-star model was found, where it is proposed to evaluate the opening of data from its accessibility and reuse through five levels, represented by stars, that evaluate: 1. If data is only published in any format under an open license, 2. If data is structured, 3. If they are in non-proprietary formats, 4. If URI is used to access specific data directly, and 5. If they are linked to other data generating context [³].

In the case of the Open Data portal of Barcelona, the authors evaluated the quality of the portal data according to its reuse, they complemented the five-star model with the proposal to include factors such as the frequency of updating and geolocation of the data and related the amount of download and themes, according to the number of stars obtained with the model [⁴]. Similar case to the evaluation of portals of the European Union, where relevance is given to the analysis of the state of the data sets and the standards in which they were published at the time for the implementation of recommendations and the general improvement of portals [⁵].

In the case of the Barcelona Open Data Portal, the authors evaluated the quality of the portal data according to its reuse, complemented the five-star model with the proposal to include factors such as the frequency of updating and geolocation of the data and related the amount of download and themes, according to the number of stars obtained with the model [⁴]. Similar case is the one of the European Union, portal evaluation, where relevance is given to the analysis of the state of the data sets and the standards in which they were published at the time for the implementation of recommendations and the general improvement of portals [⁵].

Although it is evidenced in other works that more robust portals evaluation models are proposed that complement, to a large extent, the model proposed by Berners-Lee, enriching aspects of data and portal quality [⁶], as well as the Using indicators proposed by organizations such as the Open Knowledge Foundation (OKF) for open data programs [⁷], it is considered that there are factors that are left out of the scope of the study or not covered in depth, but are necessary for the evaluation of the quality of data and portals, for example, the evaluation of metadata and communication channels offered by the portals.

As for the other studies, there are methodologies such as Meloda, which is used for the exclusive evaluation of data reuse [⁸]; the evaluation of metadata from its use, availability, completeness, openness and addressability [⁹]; the analysis of the structural composition of the portal based on its conformation and categorization [¹⁰], and the evaluation of national portals through the general characteristics of the portals and the data set [¹¹].

Among the methodologies, models and standards of found evaluation, those presented in Table 1 stand out.

Table 2 shows a consolidation of the dimensions measured by each of the methodologies described in Table 1.

III. Proposed Methodology

Taking as reference the methodologies presented in Table 1, the evaluation of open data from two approaches is proposed: 1) Published data, covering quality, use and metadata, and 2) Portal, highlighting aspects of its structure, usability and communication mechanisms. Each dimension is composed of several factors, whose general criteria are explained in Table 3.

As part of the proposed methodology, a quantitative measurement system is proposed with the objective of scoring each of the presented criteria (Table 3). Each approach, portal and data has a maximum score of 100 points, distributed as shown in Table 4. The final score will be:

𝑆𝑐𝑜𝑟𝑒=(𝑑𝑎𝑡𝑎 𝑠𝑐𝑜𝑟𝑒∗0.6)+(𝑃𝑜𝑟𝑡𝑎𝑙 𝑠𝑐𝑜𝑟𝑒∗0.4)

That is, the score obtained when evaluating the data will be equivalent to 60% of the score, and the portal score will have an equivalence of 40%. Although some of the criteria proposed may have qualitative considerations, the methodology proposes a quantitative approach to the evaluation of factors, with the objective of responding to the use of indicators to evaluate open data initiatives, as organizations such as the World Wide Web Foundation with the Open Data Barometer, or the Open Knowledge Foundation with the Global Open Data Index.

The maximum score to be obtained in each criterion that makes up each factor is presented in the boxes in Table 4. These criteria are related to the data:

In relation to the portal:

Structure:

1. Categorization: data sets are consistent with respect to similarity with other sets that are classified in the same category.

In case the score gives a decimal value, it must be adjusted by rounding. Next, Table 5 shows the scores with their corresponding classification.

If, when evaluating a portal, the data score was 50 points and that of the portal was 63 points, the following would be obtained:

𝑆𝑐𝑜𝑟𝑒=(50∗0.6)+(63∗0.4)

𝑆𝑐𝑜𝑟𝑒=30+25.2=55.2

According to the classification proposed in the methodology, the portal would have an acceptable quality.

IV. Study case “Colombia Open Data”

With the aim of evaluating the methodology, it was applied in the open data portal provided by the Colombian government (https://www.datos.gov.co/), based on the experience of a group of users, both experts as inexperienced. The qualification obtained is presented in Table 6, which also summarizes the main aspects that justify the evaluation of each factor or criterion.

All the above, gives the portal the following score:

𝑆𝑐𝑜𝑟𝑒=(48∗0.6)+(59∗0.4)

𝑆𝑐𝑜𝑟𝑒=52.4

Consequently, according to Table 5, the portal would have an acceptable rating, which indicates that, although it has different functionalities, it is necessary to add control points that provide greater satisfaction to the end user, eliminating sets that do not comply with minimum quality conditions or allowing to qualify a set by users.

V. Conclusions

The use of methodologies and models to determine the quality of the data contributes to the improvement of these, based on the identification of the status and flaws that may occur, also helps the continuation of the life cycle of open data, whose processes are in constant improvement.

Each methodology provides a different approach to the extent that its evaluation criteria is raised, which may lead to the studied element (portal or data) having different quality levels, depending on the used methodology. However, it is not unknown that the approach to a more real quality result is given by the combination and complement of methodologies and models that allow a greater number of aspects to be covered.

Open data portals play an important role in data opening initiatives, since they are the main point of access and availability of data, mainly published by government entities, which is why the quality of the data, of the structure of the data portal and the characteristics it provides to its users, can determine its level of use, impact and reputation; This is why the responsibility of the portals also lies in their constant improvement to offer users the highest possible quality.

When interacting with the Open Data portal of the Colombian State, it has been found that there are a large number of data sets available, but that many of them present inconsistencies or other flaws that hinder their use, which evidences the need to evaluate the portal with regarding its data and structure, since this type of aspects may raise the question about the use of portal resources.