SciELO - Scientific Electronic Library Online

 
vol.35 special issueBiodiversity and depressive symptoms in Mexican adults: Exploration of beneficial environmental effectsEvaluation of exposure of auto painters to organic solvents in the city of Bogota author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Article

Indicators

Related links

  • On index processCited by Google
  • Have no similar articlesSimilars in SciELO
  • On index processSimilars in Google

Share


Biomédica

Print version ISSN 0120-4157

Biomédica vol.35 no.spe Bogotá Aug. 2015

http://dx.doi.org/10.7705/biomedica.v35i0.2430 

ARTÍCULO ORIGINAL

doi: http://dx.doi.org/10.7705/biomedica.v35i0.2430

Environmental and occupational health research and training needs in Colombia: A Delphi study

Laura A. Rodríguez-Villamizar 1 , Beatriz Elena González 1 , Lina María Vera 1 , Jonathan Patz 2 , Leonelo E. Bautista 2

1 Departamento de Salud Pública, Escuela de Medicina, Universidad Industrial de Santander, Bucaramanga, Colombia

2 Department of Population Health Sciences, School of Medicine and Public Health, University of Wisconsin- Madison, Madison, WI, USA

Author´s contributions:

Laura A. Rodríguez-Villamizar and Leonelo E. Bautista were responsible for study design, data analysis, and writing of the manuscript .

Beatriz Elena González assisted in study design, design and testing of questionnaires, conducted feedback meetings, assisted in data analysis, and reviewed the manuscript.

Lina María Vera contributed to study design, interpretation of results, and review of the manuscript.

Jonathan Patz contributed to study design, interpretation of results, and review of the final manuscript.

Recibido: 02/07/14; aceptado: 06/03/15


Introduction: E nvironmental factors contribute with 16% of the burden of disease in Colombia. A main obstacle in implementing national and regional environmental and occupational health policies is the limited knowledge on the local ability to study and control the impact of harmful exposures on health.

Objective: To identify needs for research and training in environmental and occupational health in Colombia.

Materials and methods: We conducted a three-round hybrid Delphi study. A group of environmental and occupational health Colombian experts (n=16) from government agencies, universities, and research centers was recruited to participate in the study. Expert´s opinions on research and training needs were gathered through online questionnaires, followed by an in-person meeting. The percentage of agreement and the coefficient of variation were used to measure consensus.

Results: Air pollution and chemical products were considered the most important environmental and occupational exposures, due to their significant impact on chronic non-communicable diseases, such as respiratory diseases, cardiovascular diseases, and cancer. Research on the effects of outdoor air pollution on cardiovascular and respiratory diseases was considered of the greatest importance. Priority training areas included environmental and occupational health risk assessment, exposure modeling, advanced statistical methods, urban planning, occupational safety and hygiene, and epidemiology and toxicology.

Conclusions: These findings provide a valuable input for the definition and implementation of national environmental and occupational health policies and for the development of a regional hub aimed at strengthening the capacity for research and training in Colombia.

Key words: Environmental health, occupational health, research, education, Colombia.

doi: http://dx.doi.org/10.7705/biomedica.v35i0.2430


Necesidades de investigación y formación en salud ambiental y ocupacional en Colombia: un estudio Delphi

Introducción. Los factores ambientales contribuyen con el 16 % de la carga de enfermedad en Colombia. Un obstáculo importante para la implementación de políticas en salud ambiental y ocupacional es el conocimiento limitado sobre la capacidad local para estudiar y controlar el impacto de exposiciones ambientales y ocupacionales.

Objetivo. Identificar necesidades de investigación y formación en salud ambiental y ocupacional en Colombia.

Materiales y métodos. Se hizo un estudio Delphi híbrido en tres rondas. Se reclutó a un grupo de expertos en salud ambiental y ocupacional (n=16) de instituciones gubernamentales, universidades y centros de investigación. Las opiniones de los expertos sobre necesidades de investigación y formación se recolectaron mediante cuestionarios electrónicos, seguidos de una reunión presencial. El porcentaje de acuerdo y el coeficiente de variación se usaron para cuantificar el consenso del grupo.

Resultados. La contaminación del aire y los productos químicos fueron considerados por los expertos como las exposiciones más importantes, dado su gran efecto en las enfermedades crónicas no transmisibles. La investigación de los efectos de la contaminación del aire ambiental sobre las enfermedades cardiovasculares y respiratorias, se consideró de importancia máxima. Las áreas prioritarias de formación fueron la evaluación del riesgo, el modelado de la exposición, los métodos estadísticos avanzados, la planeación urbana, la higiene y la seguridad industrial, y la epidemiología y la toxicología aplicadas a la salud ambiental y ocupacional.

Conclusiones. Estos resultados son un insumo importante para la implementación de políticas nacionales en salud ambiental y ocupacional, y para el desarrollo de un nodo regional que fortalezca la capacidad de investigación y formación en Colombia.

Palabras clave: salud ambiental, salud laboral, investigación, educación, Colombia.

doi: http://dx.doi.org/10.7705/biomedica.v35i0.2430


The World Health Organization (WHO) estimates that 24% of the global burden of disease can be attributed to environmental exposures (1,2). Environmental and occupational (EO) interventions have, therefore, a large potential to improve popu lation health and reduce health disparities across the world. Yet, the role of EO interventions in reducing the global disease burden has been limited, due in part to a lack of risk assessments and cost-effectiveness studies of EO interventions in developing countries (3). In Colombia, environmental factors contribute with 16% of the burden of disease, mostly through increased risk of neuropsychiatric disorders, intentional injuries, and cancer (4). Also, in 2006 the World Bank estimated that the highest costs associated with environmental degradation and health impacts in Colombia were urban and indoor air pollution, inadequate water supply, sanitation and hygiene, natural disasters (such as floods and landslides), and land degradation (5).

Recently, environmental health issues have become a public policy priority in Colombia. In 2008, the Concejo Nacional de Política Económica y Social (CONPES) issued specific guidelines for a comprehensive environmental health policy (6). The overall objective of these guidelines was to promote integrated work on prevention and control of adverse health effects of environmental factors. Air and water quality, as well as chemical safety, were identified as the main targets for inter vention. Two specific goals defined by CONPES were: a) to promote environmental health policy actions based on population risk assessment and social determinants, and b) to strengthen the training of professionals and develop research on environmental health problems. Although CONPES´ guidelines were issued in 2008, their implementation is just under way.

One of the main obstacles in implementing sound national and regional environmental health policies is the limited knowledge on the local ability to study and control health impacts of harmful EO exposures. In response to this need, we conducted a systematic synthesis of experts´ opinions to identify gaps in EO health needs and opportunities for research and training in Colombia. The objectives of our study closely match those defined by CONPES (6). Moreover, findings from our study will be used as essential background information for the planning and development of a Global EO Health Hub in Colombia, as part of the GEOHealth Hubs Program sponsored by the US National Institutes of Health/Fogarty International Center (http:// www.fic.nih.gov/programs/Pages/environmental- occupational.aspx). This GEOHealth Hub will be aimed at strengthening the capacity for research, training, and policy formulation in South America and the Caribbean.

Materials and methods

Our systematic synthesis was based on the Delphi method, a structured group communication process used to reliably attain group consensus and make decisions based on expert judgment (7). The method was developed in the early 1950´s as an approach based on a panel of experts that can explore complex topics and develop consensus of opinion regarding future advances in a given field (8). It is used in situations where statistical or model-based procedures are not practical or possible due to a lack of adequate data (9) and it has been widely applied in social sciences and in the area of policy development to achieve convergence of opinions from experts on specific issues (10-12).

We used a collaborative or hybrid Delphi method, a modified approach involving the use of anonymous questionnaires interspersed with controlled opinion feedback during in-person meetings (13,14). This hybrid approach combined with online tools outperforms the traditional paper-based Delphi design, since reliable consensus among experts is reached in less time (15 ).

We recruited Colombian EO health experts from academic institutions and research centers, and government employees involved in EO health decision making. An initial group of 18 EO experts and stakeholders was identified by the authors and invited for a face-to-face meeting to introduce the Delphi study and our project to develop a GEOHealth Hub in Colombia. They were also asked to provide the names of other national EO experts potentially eligible to participate in the Delphi study. Through this meeting we identified 68 potential participants.

Expert selection was based mainly on their experience in EO health research, training, and policy making. The selection process was aimed at selecting 20 to 30 experts and included three sequential screenings. For the first screening we calculated an individual score that incorporated information on the type of institution the expert was associated with, and the expert´s current position, area of expertise, number of times listed as an expert by the initial group of experts, and number of indexed scientific publications in the last ten years.

For the second screening, we reviewed the curriculum vitae of the 30 experts with the highest scores in the first screening. These CVs were publicly available through the online research platform "ScienTi" prepared by the Colombian Administrative Department of Science and Tech-nology (COLCIENCIAS, http://www.colciencias.gov.co/scienti). From this source we verified and gathered further information on academic degrees, past and current research activity, and indexed and non-indexed publications.

Experts whose main field of research/work was loosely related to EO issues were excluded at this stage. For the third screening step, the authors discussed the individual information from potential participants and agreed on a list of 27 to be invited to participate. This decision was based on the expected ability of the expert to provide informed and useful opinions about the focus questions of our Delphi study and the expert´s experience on EO health. Experts received a formal invitation by electronic mail, along with a description of the study´s objective and procedures, and what was expected from them if they chose to participate. Twenty three experts accepted the invitation.

The study was developed in three consecutive rounds, using an iterative feedback process: questionnaire ® data analysis ® feedback ® new questionnaire. The results from each round were used to produce a structured questionnaire for the next round. The questionnaire design for the first round focused on these a priori questions : 1) "What are the environmental and occupational health problems that most influence disease burden in Colombia?"; 2) "How will their impact and relative importance change during the next decade?"; 3) "What are the environmental and occupational health training priorities in Colombia?", and 4) "What levels of training are needed for those training priorities?" All questionnaires were made available to participants through a username and password protected web site designed for the study. Questionnaires for rounds two and three were posted along with a summary of the findings from the previous round and an explanation of the issues to be considered in the new round. When agreement was below a predefined consensus level the issue was readdressed in an attempt to clarify and reach consensus.

The first round was aimed at identifying EO exposures with the highest impact on health in Colombia and the corresponding research priorities. We asked participants to provide a list of the five EO current agents or exposures of greatest importance in terms of their health impact and to provide a similar list for 2023, ten years from now. Also, we asked participants to list the five diseases or health outcomes and the three chronic diseases most strongly related with the five exposures they had identified as of the greatest current importance. Then, we asked participants to assign a numeric score of the potential impact of the five main exposures on their main health outcomes based on the current frequency and tendency of the exposure, the fraction of morbidity and mortality attributable to the exposure, and the feasibility of intervening to curtail the effects of the exposure. Finally, we asked participants to identify up to three research priority areas, based on their list of priority EO health problems from the previous questions. EO agents/exposures reported by participants were grouped and standardized by areas and sub-areas as suggested by Ordóñez (16). The reported diseases or he alth effects were standardized using the International Disease Classification-10 th edition (17).

The goals of the second round were: 1) to validate the findings from the first round in terms of EO exposures with the highest health impact; 2) to further specify previously identified research needs, and 3) to identify priorities in EO health training needs. We asked participants to score again the research areas identified in the previous round and gave them the opportunity to include other specific research areas they had not included the first time. Also, we invited the experts to identify the knowledge areas Colombia should have as priorities for training in EO health. For this purpose, experts were provided a hypothetical amount of money that they could distribute in direct proportion to the relative importance of each area. Then, the participants were asked to identify specific topics in which training was needed within each priority knowledge area. Finally, we requested each expert to assign the proportion of resources that should be invested in each level of training (technical or undergraduate, specialization or master, and doctorate level) for each of the specific priority training topics. Knowledge and subject areas were standardized using the National Information System for Higher Education from the Ministry of Education (18).

The main purpose of the final round was to present issues in which consensus was reached and to further discuss issues of poor consensus. This round included one in -person meeting and one on- line questionnaire administered after the meeting. Though experts surrendered their anonymity for the meeting, objectivity was preserved by keeping individual responses confidential. In this meeting the authors presented the findings from the previous rounds and encouraged an in-depth discussion of unresolved issues, i.e., those with poor consensus. During this meeting some experts called our attention to the fact that risk assessment was the dominant approach used to prioritize EO health problems in the previous rounds, and that this may have resulted in the exclusion of important problems that did not fit this approach. In response to this concern, the investigators and the group of experts decided to include anew the first component of the questionnaire used in the first round in the one used in round three, allowing the participants to depart from the risk assessment approach as they deemed fit. In this third round we also asked the experts to assign an importance score (from 0 to 100) to the seven specific subject training needs identified in the previous round. Experts also assigned independent scores to the level of training needed for those subjects.

We conducted a descriptive analysis using simple frequencies and percentages for nominal answers to open questions and calculated the total sum and the coefficient of variation (CV = standard deviation divided by the mean) for continuous scores. For open questions with nominal responses we defined consensus as a percent agreement =50%. For continuous responses we defined consensus as CV=0.5. Finally, we used the difference between the CV in two consecutive rounds as an indicator of the reliability of the degree of consensus (19).

Results

Sixteen out of 23 participants (69.6%) completed the first and second rounds, and 14 completed all three rounds (60.9%). Most of them were male (63%), 25% were from government agencies, and similar proportions (37.5%) were working in public and private universities. Ten participants worked in Bogota (10), three in Cali, and one in Bucaramanga, one in Medellín and one in Popayán. About half (56%) had a background in health sciences, and 25%, in engineering. All had graduate degrees and 63% had doctoral degrees on public health or epidemiology. Three quarters had scientific publications in national journals and about one third had more than 10 papers published in international journals.

Main EO health exposures and their health effects

Experts reached consensus in identifying chemical exposures and hazardous wastes, mainly related to contamination by heavy metals and pesticides, as well as air pollution, particularly outdoor pollution, as the most important EO exposures in Colombia at the present time (table 1). The pa rticipants also expect these exposures to be predominant in 2023, in addition to climate change which was listed as third in importance, even though consensus (38% of agreement) was below our predefined minimum. The results of the five main EO health exposures were consistent across the three rounds of the Delphi study. However, drinking water, basic sanitation, and hygiene were replaced by climate change as the fifth most important exposure iden tified in the third round. During the third round in- person meeting experts also identified other EO problems and exposures of importance, such as general and occupational accidents, exposure to solvents, psychosocial risks, environmental inequity, public policy management, zoonosis, deforestation, endocrine disruptors, biodiversity, and food safety. Nevertheless, after the third round the level of consensus about these issues was b elow our minimum.

Experts also concurred in identifying respiratory diseases, neoplasms, and cardiovascular diseases as the main types of diseases associated to the most important EO exposures (table 2). Thus, most of the health effects of priority EO exposures identified by the experts were chronic   non-communicable diseases (CNCD). However, acute respiratory, gastrointestinal diseases, and acute poisoning were also identified as important health effects.

Moreover, experts were consistent in stating that air pollution (88% agreement) and chemical and hazardous waste exposures (63% agreement) were the main exposures associated with CNCDs, followed by occupational exposures (44% agree ment). There was also consensus in that outdoor air pollution has a large impact on respiratory diseases (69% agreement). Cardiovascular diseases and lung cancer were also listed as outdoor air pollution effects, but experts did not reach minimum consensus about their importance. Experts pointed to the link between exposure to heavy metals and pesticides and malignant and benign neoplasms, but were not consistent regarding the relevance of this problem.

Research needs priorities

Upon identifying the most important EO health problems, i.e., exposures with a large absolute impact on CNCD burden, experts pointed out the three to five research topics they considered most important regarding these problems (table 3). During the second round, participants reached consensus around the idea that research on the effect of the outdoor air pollution on cardiovascular and respiratory diseases was of the greatest importance, the latter being the research topic that reached the highest consensus reliability between the first and second rounds. Experts also agreed on the need to investigate the effects of heavy metals and pesticides exposures on neurological diseases and congenital malformations. They also called attention to the need of quantifying the burden of disease attributable to exposures such as water contamination and climate change, which are hard to characterize.

Training needs priorities

Four knowledge areas were consistently identified by the experts as priority training areas: Health sciences, engineering and related areas, social/human sciences, and mathematics (table 4). The training disciplines or topics signaled by the experts as the most important were environmental and sanitary engineering, mathematical exposure models, urban planning, biology/microbiology, and EO epidemiology.

Although seven specific training topics were identified, the experts only reached minimum consensus for health risk assessment (table 5). Finally, minimum consensus on specific level of training was achieved only for mathematics-related topics (exposure models and statistical methods) at the doctoral level.

Discussion

We identified a list of EO health research and training priorities in Colombia, based on the consensus of a group of EO experts. Air pollution and chemical products were the most important EO exposures, and this was partly due to their large impact on CNCD morbidity and mortality. This finding is somehow consistent with that from the World Bank´s Country Environmental Assessment -CEA (5). Indeed, the CEA showed that two of the most costly problems associated with environmental degradation in Colombia were urban and indoor pollution and inadequate water supply, sanitation, and hygiene. In contrast, natural disasters and land degradation, the other two EO priorities identified by th e CEA were not singled out by our panel of experts. This discrepancy is likely due to the fact that our panel of experts was asked to select EO exposures based on their impact on health instead of their overall cost to society.

CNCD, particularly respiratory diseases, cardio- vascular diseases, and cancer, were identified as the main outcomes of the most important EO exposures. This was in agreement with results from the national disease burden study, which found that CNCDs accounts for 76% of total disability adjusted life years (DALYs) in both sexes in Colombia (20). In contrast, according to the WHO report on environmental health burdens, the larger fractions of disease burden attributable to environmental factors pertain to malaria and unintentional and road traffic injuries in developing countries and to cardiovascular diseases and cancer in developed countries (1). Colombia, a middle-income developing country, seems to exhibit a mixture of acute and chronic health conditions as main effects of EO exposures, with a preponderance of CNCD. This is probably explained by a transition in environmental exposures dominated by water supply and sanitation to a pattern dominated by air pollution from industrial facilities and traffic in urban settings.

We found that studies on the effects of the outdoor air pollution on cardiovascular and respiratory health, and training on EO health risk assessment, exposure modeling, EO epidemiology/toxicology and urban planning are considered of the utmost importance. As far as we know, similar assessments have not been conducted in this setting.

Fortunately, our findings are closely aligned with CONPES´ recommendation of using research findings to inform national environmental health policy focused on air and water quality, and on chemical safety (6). In fact, the fifth objective outlined in the CONPES action plan is to strengthen the research and training processes required for solving the environmental health problems in the country. Therefore, our findings about research and training gaps should inform CONPES about priorities that should be taken into account to reach this goal. They should also be useful in guiding the implementation of the research and training component of the environmental health dimension of the current Public Health Decennial Plan 2012- 2021 issued by the Colombian Ministry of Health and Social Protection (21). This plan identifies environmental health as the first of seven priority areas to address for the next decade.

We recognized that our findings are somehow subjective, as they are based on the opinions of a group of experts. Unfortunately, an assessment based on collection of hard data would have been unfeasible and would have been devoid of the perceptions and values of key players that are and will be involved in EO research, training, and policy formulation. The hybrid Delphi we used combines characteristics of directed face-to-face discussion groups, the nominal group technique, and the traditional Delphi method , maximizing their efficiency in reaching consensus (14,15).

It is also possible that our results were unduly influenced by the particular sample of experts included in our study, as this was a convenience instead of a random sample. However, our selection process was solely guided by the goal of recruiting a well qualified and minimally homogeneous group of experts to achieve valid and reliable results. In fact, the homogeneity of the group and considerable knowledge of the subject are primordial for a successful Delphi study, even more than group size. Also, the number of participants in our study (n=16) was within the ideal range (10 to 18) for a Delphi panel (22).

Although it is possible that expert´s opinions depended on their areas of research or work, the iterative feedback process using summary findings of previous rounds to re-assess needs should have limited this potential bias. Also, we cannot discard that experts were more prone to identify as more important those research topics with large gaps in knowledge, but with limited impact on EO health. However, to minimize this bias, experts were asked to assess the importance of exposure/disease combinations by considering the prevalence of the exposure and its relative and absolute impact on health, in addition to local gaps in knowledge . It is also po ssible that expert´s identification of research priorities was based on well documented EO health problems and, in consequence, other important research needs related to less well known problems might have been missing from the research priority list.

In summary, identifying EO health research and training needs is only one of the first steps in achieving the goal of defining a comprehensive national EO health policy. Findings from this study provide valuable input for the implementation of national environmental health policies and for the planning and development of a regional GEOHealth hub aimed a t strengthening the capacity for EO health research, training, and policy formulation in the region.

Acknowledgements

Authors thank all experts for their committed participation. We also want to thank Sergio Serrano for his collaboration in the programming of the online questionnaires.

Conflicts of interest

No conflicts of interest to declare.

Funding

This study was supported by the US National Institutes of Health/Fogarty International Center grant No. 1R24TW009567-01/02.

Corresponding author: Laura A. Rodríguez-Villamizar, Carrera 32 N° 29-31, oficina 301, Bucaramanga, Colombia Phone: (577) 645 4726; fax: (577) 635 1646 laurovi@uis.edu.co, laurarovi78@gmail.com

References

1. Prüss-Üstün A, Corvalán C. Preventing disease through healthy environments towards an estimate of the environmental burden of disease. Geneva: WHO; 2006.         [ Links ]

2. Prüss-Ustün A, Bonjour S, Corvalán C. The impact of the environment on health by country: A meta-synthesis. Environ Health. 2008;7:7. http://dx.doi.org/10.1186/1476-069X-7-7         [ Links ]

3. Prüss-Üstü A, Corvalán C. How much disease burden can be prevented by environmental interventions? Epidemiology. 2007;18:167-78. http://dx.doi.org/10.1097/01.ede.0000239647.26389.80         [ Links ]

4. World Health Organization. Country profiles by environ- mental burden of disease. Geneva: WHO; 2009. p. 35.         [ Links ]

5. World Bank. Colombia - Mitigating environmental degradation to foster growth and reduce inequality. Washington, D.C.: World Bank; 2006. p. 388.         [ Links ]

6. Consejo Nacional de Política Económica y Social, República de Colombia, Departamento Nacional de Planeación. Documento CONPES 3550. Lineamientos para la formulación de la política integral de salud ambiental con énfasis en los componentes de calidad de aire, calidad de agua y seguridad química. Bogotá, D.C.; CONPES; 2008. p. 54.         [ Links ]

7. Landeta J. El método Delphi: una técnica de previsión del futuro. Barcelona: Editorial Ariel Social; 2002.         [ Links ]

8. Ono R, Wedemeyer D. Assesing the validity of the Delphi technique. Futures. 1994;26:289-304. http://dx.doi.org/10.1016/0016-3287(94)90016-7         [ Links ]

9. Rowe G, Wright G. The Delphi technique as a forecasting tool: Issues and analysis. Int J Forecast. 1999;15:353-75. http://dx.doi.org/10.1016/S0169-2070(99)00018-7         [ Links ]

10. Gupta UG, Clarke RE. Theory and applications of the Delphi technique: A bibliography (1975–1994). Technol Forecast Soc Change. 1996;53:185-211. http://dx.doi.org/10.1016/S0040-1625(96)00094-7         [ Links ]

11. Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Change. 2006;73:467-82. http://dx.doi.org/10.1016/j.techfore.2005.09.002         [ Links ]

12. Hsu C, Sandford B. The Delphi Technique: Making sense of consensus. Pract Assessment, Reserch & Evaluation. 2007;12:1-8.         [ Links ]

13. Rowe G, Wright G. The Delphi technique: Past, present, and future prospects — Introduction to the special issue. Technol Forecast Soc Change. 2011;78:1487-90. http://dx.doi.org/10.1016/j.techfore.2011.09.002         [ Links ]

14. Landeta J, Barrutia J, Lertxundi A. Hybrid Delphi: A method- ology to facilitate contribution from experts in professional contexts. Technol Forecast Soc Change. 2011;78:1629-41. http://dx.doi.org/10.1016/j.techfore.2011.03.009         [ Links ]

15. Gordon T, Pease A. RT Delphi: An efficient, "round-less" almost real time Delphi method. Technol Forecast Soc Change. 2006;73:321-3. http://dx.doi.org/110.1016/j.techfore.2005.09.005         [ Links ]

16. Ordóñez GA. Salud ambiental: conceptos y actividades. Rev Panam Salud Pública. 2000;7:137-47. http://dx.doi.org/10.1590/S1020-49892000000300001         [ Links ]

17. World Health Organization. International Disease Classification ICD-10. 2010. Fecha de consulta: 7 de junio de 2013. Disponible en: http://apps.who.int/classifications/icd10/browse/2010/en.         [ Links ]

18. Jaramillo H. Hacia un Sistema Nacional de Información de Educación Superior. Bogotá, D.C.: Universidad del Rosario; 2005.         [ Links ]

19. von der Gracht H. Consensus measurement in Delphi studies. Technol Forecast Soc Change. 2012;79:1525-36. http://dx.doi.org/10.1016/j.techfore.2012.04.013         [ Links ]

20. Acosta-Ramírez N, Peñalosa R, Rodríguez-García J. Carga de enfermedad Colombia 2005: resultados alcanzados. Centro de Proyectos para el Desarrollo (CENDEX) de la Pontificia Universidad Javeriana. Documento técnico ASS/1502-08. Bogotá, D.C.: Pontificia Universidad Javeriana; 2008. p. 1-94.         [ Links ]

21. Ministerio de Salud y Protección Social. Plan Decenal de Salud Pública. Bogotá; Ministerio de Salud y Protección Social; 2013. p. 237.         [ Links ]

22. Okoli C, Pawlowski SD. The Delphi method as a research tool: An example, design considerations and applications . Inf Manag 2004;42:15-29. http://dx.doi.org/10.1016/j.im.2003.11.002         [ Links ]