SciELO - Scientific Electronic Library Online

 
vol.15 número1ISOTERMAS Y CALOR ISOSTÉRICO DE ADSORCIÓN DE AGUA DE ALMIDÓN DE QUINUAANÁLISIS DE LAS FUERZAS DEL MERCADO DENDROENERGÉTICO EN LA VEREDA CAJETE DEL MUNICIPIO DE POPAYÁN índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Indicadores

Links relacionados

  • En proceso de indezaciónCitado por Google
  • No hay articulos similaresSimilares en SciELO
  • En proceso de indezaciónSimilares en Google

Compartir


Biotecnología en el Sector Agropecuario y Agroindustrial

versión impresa ISSN 1692-3561

Rev.Bio.Agro vol.15 no.1 Popayán ene./jun. 2017

https://doi.org/10.18684/BSAA(15)105-115 

DOI: http://dx.doi.org/10.18684/BSAA(15)105-115

MÉTODO DELPHI EN ESTUDIOS DE PROSPECTIVA TECNOLÓGICA: UNA APROXIMACIÓ PARA CALCULAR EL NÚMERO DE EXPERTOS Y APLICACIÓN DEL COEFICIENTE DE COMPETENCIA EXPERTA 'K'

DELPHI METHOD IN TECHNOLOGICAL FORESIGHT STUDIES: AN APPROACH TO CALCULINTING THE NUMBER OF EXPERTS AND THE APPLICATION OF THE COMPETENCE COEFFICIENT 'K' EXPERT

MÉTODO DELPHI EM ESTUDOS DE PROSPECTIVA TECNOLÓGICA: UMA APROXIMAÇÃO PARA CALCULAR O NÚMERO DE EXPERTOS E APLICAÇÃO DO COEFICIENTE DE COMPETÊNCIA EXPERTA 'K'

JHON WILDER ZARTHA-SOSSA1, JUAN MANUEL MONTES-HINCAPIÉ2, IVAN DARIO TORO-JARAMILLO3, RAUL HERNANDEZ-ZARTA4, HECTOR SAMUEL VILLADA-CASTILLO5, JOSE LUIS HOYOS-CONCHA6


1 Universidad Pontificia Bolivariana, Facultad de ingenierias. Msc. Gestión tecnológica. Medellin, Colombia.

2 Universidad de Medellín, Facultad de Ciencias Económicas y Administrativas. Ph.D. Proyectos de Innovación Tecnológica en la Ingeniería de Producto y Proceso. Medellin, Colombia.

3 Universidad de Medellín, Facultad de Ciencias Económicas y Administrativas. Ph.D. Filosofía. Medellin, Colombia.

4 Universidad Pontificia Bolivariana. Facultad de ingenierias. Esp. Gestión y Desarrollo Agroindustrial. Medellin, Colombia.

5 Universidad del Cauca, Facultad de Ciencias Agrarias. Ph.D. Ingeniería Con Enfasis En Alimentos. Popayán, Colombia.

6 Universidad del Cauca, Facultad de Ciencias Agrarias. Ph.D. Ingeniería Con Enfasis En Alimentos. Popayán, Colombia.

Correspondencia: jhon.zartha@upb.edu.co


Recibido para evaluación: 13 de Julio de 2016. Aprobado para publicación: 14 de Diciembre de 2016.



ABSTRACT

This paper deals with the Delphi method, its origins and characteristics. The main aspect to address is to provide information to identify an adequate number of experts in the choice of topics in the applications of the Delphi method in foresight. For that purpose, one Delphi study was evaluated in the period 2012 - 2015 on priority technologies regarding biodegradable packaging in the year 2032. On the results of this study, the incidence of the number of experts was evaluated with 9, 15, 22 and 24 experts. The results highlighted the existence of common core of technologies that were priorities from analysis 9 to 24 experts; regarding the expert competence coefficient 'K' the results with the 16 experts with a coefficient 'k' greater than or equal to 0.8, in four of the five groups, achieved high agreement percentages in relation to the core group of priority topics.

KEYWORDS: Future studies, Number of participants, Biodegradable packaging.


RESUMEN

Este artículo trata sobre el método Delphi, sus orígenes y características. El principal aspecto a abordar es el de proveer información para identificar el número adecuado de expertos en la elección de temas en las aplicaciones del método Delphi en prospectiva. Para este propósito un estudio Delphi fue evaluado en el periodo 2012-2015 en tecnologías prioritarias relacionadas con empaques biodegradables al año 2032. Sobre los resultados de este estudio la incidencia del número de expertos fue evaluada con 9, 15, 22 y 24 expertos. Los resultados muestran la existencia de un núcleo común de tecnologías que fueron prioritarias desde el análisis con 9 a 24 expertos; con relación al coeficiente de competencia experta 'K' los resultados con los 16 expertos con coeficiente 'K' mayor o igual a 0.8, en cuatro de los cinco grupos lograron porcentajes de acuerdo altos con relación al grupo medular de temas prioritarios.

PALABRAS CLAVE: Estudios de futuro, Número de participantes, Empaques biodegradables.


RESUMO

Este articulo trata sobre o método Delphi, suas origines e características. O principal aspecto a estudar é o fornecedor de informação para identificar o número adequado de expertos na eleição de temas, tecnologias ou variáveis nas aplicações do método. Para isto, foram avaliados Um estudo Delphi no período 2012-2015 em aspectos e tecnologias prioritárias relacionadas com embalagens biodegradáveis ao ano 2032, A partir dos resultados obtidos, foi avaliada a incidência do número de expertos com 9, 15, 22 e 24 expertos. Os resultados mostram a existência de um núcleo comum de tecnologias que foram prioritárias desde a analises com 9 a 24 expertos com relação ao coeficiente de competência experta 'K' os resultados com os 16 expertos com coeficiente 'K' maior ou igual a 0.8, em quatro dos cinco grupos lograram percentagens de acordo altos com relação ao grupo medular de temas prioritários.

PALAVRAS-CHAVE: Estudos de futuro, Número de participantes, Embalagens biodegradáveis.


INTRODUCTION


This article is part of a PhD thesis related to the modified Delphi method as a driving force for innovation strategy in the framework of an innovation management model in productive sector organizations.

The main purpose is to provide evidence to establish an appropriate number of experts for application of the Delphi method. To this end, one Delphi study evaluated in the period 2012 - 2015 on priority issues and technologies in biodegradable packaging in the year 2032, the project is part of a component in a larger study on biodegradable packaging and was supported by the National Planning Department of Colombia, the general system of royalties, governor of Cauca and the University of Cauca.

Specifically what is sought is to provide inputs for decision making with the appropriate number of experts and if a small number of experts is enough to make a Delphi study. Additionally, this article presents an application of expert competence coefficient 'k', its interpretation and the group's relationship with a higher coefficient 'k' with the issues presented as results from the study with the total number of experts.

The article is divided into a variety of sections: Introduction, theoretical framework on the Delphi method, the modified Delphi and competence coefficient 'k' expert, as well as the methodology, results, discussion and conclusions.


Theoretical frame

The Delphi method had its beginnings between 1940 and 1960 [1], according to [2] its origin is attributed to the RAND - Research and Development Corporation, in the late 40s, Santa Monica - United States. The method originated with a study of the Cold War American industrialists to identify potential targets and their vulnerability to Soviet ammunition [3], according to [4] the Delphi is a method that has been used in national technology foresight exercises, and began by being used by Japan in their five-year foresight.

The original method, instead of using the traditional approach to reaching consensus through open discussion, completely eliminates the activity of a committee, which reduces the influence of certain psychological factors such as persuasion, the unwillingness to abandon publicly expressed opinions and the contagion effect of the majority of opinion.

This technique replaces direct debate by a carefully designed program of sequential individual interrogations through interspersed information and feedback derived by consensus, calculated from previous rounds of questionnaires.


Comparison between applications of the Delphi Method

International applications have been made from the Delphi method in various sectors; presented below in Table 1, is the analysis of applications made by 14 authors of scientific publications: Also, there have been other studies related, for example, opinions Delphi experts facing the assessment of factors related to supply chain companies in Taiwan [17], application in the framework a foresight project in the textile industry in Poland [18], Characteristics and functions for place brands [19], analysis of forest fires causes and their motivations in northern Algeria [20] a patient centered early warning system to prevent readmission after colorectal surgery [21] and identification the baseline food safety training practices for retail delis using the Delphi expert consensus method [22].

Table 1


Other applications of the Delphi method can be found in innovation public policies. Such is the case of the OCDE studies regarding the 'technology push' stimulus, meaning supply through R+D public investment and incentives to companies that are considered innovative [23]. Other studies can be found about a new focus of innovation management, approaching it from a more articulated manner that involves a variety of mechanisms for diffusion and ownership of knowledge in function to the impact on the rate, the course of innovation and the technical change [24] In the same line lies the study of [25] on demand based policies, where the result of these is affirmed, bowed to the context and an emphasis is made towards well-articulated objectives.

Various studies have been made in Colombia, the most recent ones are qualitative work related foresight studies in the construction sector, whose objective was to identify the future necessities of the human resources that will arise from the impact of technological and organizational tendencies. These tendencies will diffuse in the construction sector in Colombia in the next 5 to 10 years [26]. The research methodology proposed by a guide of clinical prevention, diagnostic and treatment of acute diarrhea in children under 5 years old is a critical analysis of evidence, related to the opinion of a group of experts [27]. The study proposed a guide for the performance of panels of experts on transferable risk assessment in projects linked to the public and private sectors [28].

In the table 1, it can be seen that there is no consensus on the number of experts who answer the Delphi rounds; some authors performed their studies with high values, such as: 30, 32, 37, 39, 57, 65, 67, 68 and 123 experts, while other authors have suitable experts as figures such as: 7, 9, 10, 11, 12, 14, 15, 18, 20 or 24. This type of gaps related to the number of experts continues, as even for some researchers there is no evidence that increasing the panel size will improve accuracy. Some authors argue that ranging from 5 to 20 people is sufficient [29].

Expert competence coefficient 'K' The calculation of the 'expert competence coefficient' is made from the opinion rendered by an expert on a high level of knowledge about a research problem, as well as the sources that allow arguments on the established criteria [30].

The coefficient is obtained by applying the following formula: K = ½ (Kc+Ka), where: Kc = is the 'coefficient of knowledge' or information that the expert on the subject or problem posed. It is calculated from the assessment made by the expert himself on a scale of 0 to 10, multiplied by 0,1.

Ka = this is called the 'coefficient of argument' or foundation of expert opinion. It is obtained from assigning scores to a number of different sources of argument that could wield the expert. In Table 2, the scores are usually used for assessing the sources of arguments [30].

Table 2


With the final values obtained experts are classified into three groups, if K is greater than 0,8, greater than or less than or equal to 1: then there is high influence from all sources, if K is greater than or equal to 0,7, or greater than or equal to 0,8: then there is no influence average from all sources, now, if K is greater than or equal to 0,5, or greater than or equal to 0,7 then there is little influence from all sources [19].

According to guidelines [19] experts with values less than 0,8 are not included in the study and therefore are rejected.


METHOD


Phase I

Were reviewed scientific articles based on scientific data, theses and specific bases in leading universities in the Delphi method. Also, patent databases and technical documents related to the method and the problem were identified to be treated in this case related to the number of experts who answer a round Delphi.

This aspect is important since several studies conducted in recent years have had the problem of achieving a large number of experts, and even the invitations are a number of approximately 100 experts, only a small percentage of these rounds were answered [32, 33, 34].

Quantities of experts were used in the cases presented in the papers established and some of the figures related to the lowest values were selected, in this case: 9, 15, 22 and the final number of experts. The quantity of 7 experts was not elected for analysis because this multiple number of values of the same mode theme, such as technology or variable is presented.


Phase II

One Delphi study in the period 2012-2015 was evaluated on issues, technologies and priorities variables in biodegradable packaging in 2032.

As for the choice of topics, technologies and priority, criteria variables were held constant, which consists in calculating mode, modal frequency and percentage of consensus on these data. The average percentage of agreement was determined for each group, based on this information, the following criteria was established: Theme, technology or priority variable, consensus group above average and mode of 4 or 5, the scale was 0-5 where the order of priority was: 0 = No priority. 1 = Very Low. 2 = Low. 3 = Media. 4 = High. 5 = Very High. N = Do not know / No answer


Phase III

Percentages of common topics or technologies in each of the courts, in order to show whether the issues or technologies effectively remained constant or changed when the number of experts that were established was higher.


Phase IV

Expert coefficient competence 'K' was calculated, which is considered a measure of the level of competence in a component panel of experts, and in this sense, can be useful to obtain the coefficient of competence proposed [31].

The methodology for calculating the coefficient comprises on calculating two coefficients: ka and kc, where kc is the 'coefficient of knowledge' or owned expert information about the problem or issue raised; and a self-assessment on a scale of graphic seal (0- 10) and multiplied by 0,1. [31] In Table 2, the form of rating sources to obtain the 'Argument coefficient' (Ka) required is presented for finding the expert competence coefficient 'k' Issues and technologies that had been shown as a final result of the Delphi study, the total number of experts, topics and technologies that had been a priority for the group coefficient 'k' were finally compared.

Greater than 0.8 is considered high and for the group of coefficient 'k' low is less than 0.8.


RESULTS


Delphi Biodegradable Packaging.

A structured and anonymous survey was done to experts in Colombia and other countries such as France, Spain, Mexico, Italy and Argentina, on technological innovations, raw materials for the production of biodegradable polymer materials, raw materials for the production of biodegradable packaging, and additives to the year 2032 via a digital questionnaire of 198 items classified in 5 group of technologies. The survey was answered by experts; these results are shown in Table 3.

Table 3


In the group 'TECHNOLOGICAL INNOVATIONS', we can see that there is a common core of 5 topics that recur in the 4 groups of responses. In the group 'RAW MATERIALS FOR THE PRODUCTION OF BIODEGRADABLE POLYMERS', we can see that there is a core of 6 common topics that recur in the 4 groups of responses.

In the group 'RAW MATERIALS TO OBTAIN BIODEGRADABLE PACKAGING', we can see that there is a common core of 7 topics that recur in the 4 groups of responses. In the consolidator 'ADDITIVES', we can see that there is a common core of one recurring theme in the 4 groups of responses. Finally, in the group 'TECHNOLOGY', we can see that there is a common core of six topics that recur in 4 groups of responses.


Expert competence coefficient 'k'

The following Table shows the calculation of the coefficient 'k' to the experts of the Delphi study of biodegradable packaging. The presented information was obtained from the information requested from the nominees, about their self-worth in terms of degree of knowledge on the subject and level of argumentation. According to the calculation of the coefficient 'k', 16 experts obtained a higher value of 0,8. With this information we proceeded to analyze the results obtained by this new group of experts.

Table 4


In the group of Technological Innovations, with 9 answers, a common core of 5 topics plus 8 different topics in relation to 22 experts was obtained. Along with 15 responses, he obtained a common core of 5 topics plus 4 different topics in relation to 22 experts, In this case, the results were acquired with less variation of the 15 experts.

In the group of Raw Materials for the Production of Biodegradable Polymers, with 9 answers, a common core of 6 topics plus 2 different topics related to 22 experts. Along with 15 responses, a common core of 6 topics was obtained plus 1 topic varying differently to 22 experts, again with less variation of 15 experts was obtained.

In the group of Raw Materials for the Production of Biodegradable Packaging, with 9 answers, a common core of 8 topics plus 4 different topics in relation to 22 experts. Along with 15 responses, a common core of 8 topics plus 0 subjects was achieved relative to 22 various experts. This time the same end results were acquired with 15 experts, that is, there was no variation between the final answers reported by the 22 experts compared to 15 experts.

In the group of additives, with 9 answers, a common core of 1 topic plus 3 different topics were achieved in relation to 22 experts. When 15 responses were obtained, a common core of 1 item plus 1 different topic was acquired in comparison to the 22 experts. Again the least variation was presented with the 15 experts.

In the group of technologies, with 9 answers, a common core of 6 topics plus 1 different theme was achieved in comparison to 22 experts. Along with 15 responses, a common core of 6 topics plus 1 different theme was obtained compared to 22 experts. In this particular case, the same results with exper ts 9 and 15 compared with the final results reported. According to the analysis in the Delphi study, it can be observed that the value of experts with less variation are 15 experts to study biodegradable packaging.

In the latter, the number of 15 experts scored second in terms of less variation in the results, this is consistent with the comments made by several authors. For example [14], applied the Delphi method for the selection of appropriate experts and detailed principles for decision-making during the process to ensure the validity of a Delphi study. They mention an example in e-commerce in Sub-Saharan Africa that used 10 to 18 experts for each panel, and there were 4 panels in total.

On the other hand [35], presented their results with only 14 and 12 experts in order to reach consensus among a panel of experts. The results were used to establish the best clinical criteria for the diagnosis of carpal tunnel syndrome.

[8], in the study of prognostic variables of financialmarket in a controlled experiment used only 10 experts. In this same study cited back to [29], it mentions that ten people are enough and that there is not evidence to suggest increasing the size beyond 7 to 10 people improves accuracy.


Results obtained with expert competence coefficient 'k'

The results about the appropriate number of experts were completed by the calculation of the coefficient 'k' for the study of biodegradable packaging. The results achieved with the 16 experts presented that when coefficient 'k' is greater than 0,8 there is a higher percentage of agreement with the final results.

With the results given by the experts, it is presented that when coefficient 'k' is lower there is a higher variation when compared to the final results. Furthermore, these 'less expert' have risen to the level of 'priority' topics than that of the final results that arrived at the level of 'discussion' topics. If this study had been conducted with 16 experts with a higher 'k', the 'most experts' would have obtained virtually the same results as with the final 22 experts.

The results of percentage of matches are shown in Table 5.

Table 5


Experts with a lower coefficient 'k' in the grouping 'Technological innovation' presented that the priority topics are those that the total number of experts had determined as 'discussion'. That means, when compared to the final results there is an increase of 63,63 in the priority topics, compared to only 9,09% of experts who reported changes with a higher coefficient 'k'. In the grouping 'premiums for the Production of Biodegradable Polymer materials', experts with a lower coefficient 'k' presented an increase of 100% in the priority topics, as compared to 22% who reported changes in a higher coefficient 'k'. Within the grouping 'Raw Materials for the Production of Biodegradable Packaging', experts with a lower coefficient 'k' saw an increase of 80% in the priority topics, as compared to 40% who reported changes in the experts with a higher coefficient 'k'. In the grouping 'Additives', experts with a lower coefficient 'k' saw an increase by 120% in the priority topics, versus 0% of experts who reported changes in a higher coefficient 'k'. In the grouping 'Technologies', experts with a lower coefficient 'k' presented a 46,67% increase in the priority topics, as compared to 0% of the reported changes experts with a higher coefficient 'k'.


CONCLUSIONS


According to the results of the study, it can be observed that regardless of the number of experts, a common core of subjects is presented. In this perspective, although the study had been done with a small number of experts, it could have had medullary winning topics.

Uncertainty about the appropriate number of experts and the desirability of increasing the number may have been reduced because there is not enough evidence that increasing the number of experts would obtain different results to those obtained with 15 experts.

As for the expert competence coefficient 'k' in the Delphi study on biodegradable packaging, the results obtained with the 16 experts with a coefficient 'k' greater than or equal to 0,8, in four of the five highest percentages of agreement groups were achieved with the common core of priority topics, while experts with a coefficient less than 0,8 in the five aggregators 'k' showed percentages of high variation given the same common core subjects. This means that 'less expert' highlighted other issues that were not priorities for the group of experts and the common core. According to this study, it is better to work with a small number of experts with a high coefficient 'k'.

The main limitations of the work carried out in this article has to do with the need to perform the analysis with a larger number of experts, and verify whether other applications of the method in a common core of subjects that do not change when the increase occurs in the number of experts; also, if this common set of issues is the same as mentioned by the group of experts with highest coefficient k.



REFERENCES

[1] NIELSEN, C. and THANGADURAI, M. Janus and the Delphi oracle: Entering the new world of international business research. Journal of International Management, 13, 2007, p. 147-163.         [ Links ]

[2] LANDETA, J. Current validity of the Delphi method in social sciences. Technological Forecasting and Social Change, 73(5), 2006, p. 467-482.         [ Links ]

[3] FLETCHER, A. and MARCHILDON, G. Using the Delphi Method for Qualitative, Participatory Action Research in Health Leadership. International Journal of Qualitative Methods, 13(1), 2014, p. 1-18.         [ Links ]

[4] CASTELLÓ, M. y CALLEJO, J. La prospectiva tecnológica y sus métodos. Dirección nacional de planificación y evaluación [online]. 2000. Disponible en http://www.oocities.org/es/derviscorro/plan/05.pdf [Citado 12 de octubre de 2015]         [ Links ].

[5] CHUANG, Y., CHIA, S. and WONG, J. Supply Chain Alliance Factors Evaluation by the Delphi- AHP Method. Wseas transactions on business and economics, 10(2), 2013, p. 69-79.         [ Links ]

[6] CODURAS, A. y SEÑARÍS, J. Análisis Prospectivo Delphi Sobre el Futuro de la Prestación Farmacéutica. Fundación Gaspar Casal - Fundación Aztrazeneca [online]. 2009. Disponible en http://www.fgcasal.org/publicaciones/delphi.pdf 2009 [Citado 20 de octubre de 2015]         [ Links ].

[7] WAKEFIELD, R. and WATSON, T. Delphi 2.0: A reappraisal of Delphi method for public relations research. Public Relations Review, 40(3), 2013, p. 577-584.         [ Links ]

[8] KAUKO, K. and PALMROOS, P. The Delphi method in forecasting financial markets— An experimental study. International Journal of Forecasting, 30(2), 2014, p. 313 - 327.         [ Links ]

[9] CANCELO, M., NEYRO, J. y BAQUERO, J. Tratamiento adyuvante de la vaginitis con probióticos. Progresos de Obstetricia y Ginecología, 57(1), 2014, p. 4-13.         [ Links ]

[10] CABERO, J. Formación del profesorado universitario en TIC - Aplicación del método Delphi para la selección de los contenidos formativos. Educación XX1, 17(1), 2013, p. 112-130.         [ Links ]

[11] LIIMATAINEN, H., KALLIONPÄÄ, E., PÖLLÄNEN, M., STENHOLM, P., TAPIO, P. and MCKINNON, A. Decarbonizing road freight in the future - Detailed scenarios of the carbon emissions of Finnish road freight transport in 2030 using a Delphi method approach. Technological. Technological Forecasting and Social Change, 81(1), 2013, p. 1- 40.         [ Links ]

[12] DEBIN, M., SOUTY, C., TURBELIN, T., BLANCHON, T., BOËLLE, P., HANSLIK, T., HEJBLUM, G., LE STRAT, Y., QUINTUS, F. and FALCHI, A. Determination of French influenza outbreaks periods between 1985 and 2011 through a web-based Delphi method. BMC Medical Informatics and Decision Making, 13(138), 2013, p. 1472-6947.         [ Links ]

[13] CREANGE, A. and CAREYRON, A. The diagnosisof chronic inflammatory demyelinating polyneuropathy: A Delphi-method approach. Journal of Neurology. 260(12), 2013, p. 3015-3022.         [ Links ]

[14] OKOLI, C. and PAWLOWSKI, S. The Delphi method as a research tool: An example, design considerations and applications. Information and Management, 42(1), 2004, p. 15-29.         [ Links ]

[15] GRAHAM, B., REGEHR, G. and WRIGHT, J. Delphi as a method to establish consensus for diagnostic criteria. Journal of Clinical Epidemiology, 56(12), 2003, p. 1150-1156.         [ Links ]

[16] GORDON, T. and PEASE, A. RT Delphi: An efficient, bround-lessQ almost real time Delphi method. Technological Forecasting and Social Change, 73, 2006, p. 321-333.         [ Links ]

[17] MOORE, A. A modified Delphi Study of Leadership Profiles of the New Product Development Process Stages. [Ph.D. Thesis Doctor of Management] Seattle (United States): University of Phoenix, 2001, 155 p.         [ Links ]

[18] WYSOKINSKA, Z., KOSZEWSKA, M., CZAJKOWSKIT, T. and OLSZOWY, M. Future of the Polish Textile Industrial Sector. An Overall Analysis of the Empirical Research Performed with the Delphi Method within the Project Foresight Modern Technologies for the Textile Industry. A Chance for Poland. Fibres and Textiles in Eastern Europe, 21(4), 2013, p. 10-15.         [ Links ]

[19] VELA, E., CAVIA, F., NOGUE, J. and MORALES, M. Characteristics and functions for place brands based on a Delphi method. Revista Latina de Comunicación Social. 68, 2013, p. 656-675.         [ Links ]

[20] SAHAR, O., MEDDOUR, R., LEONE, V., LOVREGLIO, R. and DERRIDJ A. Analysis of forest fires causes and their motivations in northern Algeria: the Delphi method. Biogeosciences and Forestry, 6(5), 2013, p. 247-254.         [ Links ]

[21] LI, L., MILLS, W., GUTIERREZ, A., HERMAN, L., BERGER, D. and NAIK, A. A Patient-Centered Early Warning System to Prevent Readmission after Colorectal Surgery: A National Consensus Using the Delphi Method. Journal of the American College of Surgeons, 216(2), 2013, p. 210-216.         [ Links ]

[22] KIM, K., CORLISS, A., O'BRYAN., PHILLIP, G., CRANDALL., STEVEN, C., RICKE, JACK, A. and NEAL JR. Identifying baseline food safety training practices for retail delis using the Delphi expert consensus method. Food Control, 32(1), 2013, p. 55-62.         [ Links ]

[23] Organización para la Cooperación y el Desarrollo Económico (OCDE). Promoviendo el desarrollo de sistemas locales de innovación. El caso de Medellín Colombia. Medellin (Colombia): 2015.         [ Links ]

[24] Organización para la Cooperación y el Desarrollo Económico (OCDE). Cuestionario OCDE - Lista de información necesaria Para el informe de antecedentes. Apoyo al desarrollo de sistemas de innovación: El caso de Medellín y Colombia. Medellin (Colombia): 2013.         [ Links ]

[25] EDLER, J. Review of policy measures to stimulate private demand for innovation. Concepts and effects. Compendium of Evidence on the Effectiveness of Innovation Policy Intervention, Nesta Working Paper, 13(13), 2013, p. 1-40.         [ Links ]

[26] COLOMBIA. MINISTERIO DE TRABAJO. Prospectiva laboral cualitativa para el sector construcción de edificaciones en Colombia. Bogotá (Colombia): 2015, 130 p.         [ Links ]

[27] COLOMBIA. MINISTERIO DE SALUD Y PROTECCIÓN SOCIAL - COLCIENCIAS. Guía de práctica clínica para prevención, diagnóstico y tratamiento de la enfermedad diarreica aguda en niños menores de 5 años. Bogotá (Colombia): 2013, 236 p.         [ Links ]

[28] COLOMBIA. DEPARTAMENTO NACIONAL DE PLANEACIÓN (DNP). Guía para la realización de paneles de expertos en la valoración de riesgos transferibles en proyectos de asociación público privada. Bogotá (Colombia): 2014, 16 p.         [ Links ]

[29] ROWE, G. and WRIGHT, G. Expert opinions in forecasting: the role of the Delphi technique. International journal Armstrong, Principles of forecasting, 2001, p. 125-144.         [ Links ]

[30] CABERO, J. y BARROSO, J. La Utilización del Juicio de Experto para la Evaluación de Tic: El Coeficiente de Competencia Experta. Bordón, 65(2), 2013, p. 25-38.         [ Links ]

[31] ROMERO, R., CABERO, J., LLORENTE, M. y VÁZQUEZ, A. El método Delphi y la formación del profesorado en TIC. Universidad de Sevilla [online]. 2011. Disponible en https://idus.us.es/xmlui/bitstream/handle/11441/24653/file_1.pdf?sequence=1&isAllowed=y. [citado el 14 de 2015]         [ Links ].

[32] ZARTHA, J., MONTES, J., TORO, I. y VILLADA, H. Método Delphi - Propuesta para el cálculo del número de expertos en un estudio Delphi sobre empaques biodegradables al 2032. Revista Espacios, 35(13), 2014, p. 10-11.         [ Links ]

[33] HERNANDEZ, R., VILLADA, H., ZARTHA, J., ARANGO, B., GÓMEZ, R., WALTEROS, L., DELGADO, K., MONTILLA, C., VARONA, G., MORENO, J., OROZCO, G. y PALACIO, J. Vigilancia tecnológica y análisis del ciclo de vida de la tecnología: evaluación del potencial comercial de un prototipo de guantes biodegradables a partir de almidón termoplástico de yuca. Revista Espacios, 37(13), 2016, p. 27-28.         [ Links ]

[34] ZARTHA, J., VILLADA, H., HERNANDEZ, R., FERNANDEZ, A., ARANGO, B., OROZCO, G., BERMUDEZ, R., JOAQUI, D., CERON, A. and MORENO, J. Aplication of Delphi Method in a foresigth study on biodegradable packaging up to 2032. Revista Espacios, 36(15), 2015, p. 3-4.         [ Links ]

[35] GRAHAM, B., REGEHR, G. and WRIGHT, J. Delphi as a method to establish consensus for diagnostic criteria. Journal of Clinical Epidemiology, 56(12), 2003, p. 1150-1156.         [ Links ]

Creative Commons License Todo el contenido de esta revista, excepto dónde está identificado, está bajo una Licencia Creative Commons