Application of the Delphi Method for the inclusion of externalities in occupational safety and health analysis

Aplicación del método Delphi para la inclusión de las externalidades en análisis de seguridad y de salud laboral

Delfina Ramos ^ab, Pedro Arezes ^b & Paulo Afonso ^b

^a Technology School, Polytechnic Institute of Cávado and Ave, Barcelos, Portugal gramos@ipca.pt
^b Center Algoritmi, University of Minho, Guimarães, Portugal. parezes@dps.uminho.pt; psafonso@dps.uminho.pt

Received: November 30^th, 2015. Received in revised form: February 29^th, 2016. Accepted: March 07^th, 2016.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Abstract
Organizations should regularly conduct an assessment of their occupational hazards in order to design and implement preventive measures that are necessary and sufficient to deal with the level of risk, the costs of prevention and the safety at levels considered acceptable by the organization. Furthermore, the selection of measures to be implemented in an organization should take into account both internal and external costs. Externalities are of great importance in terms of the costs of accidents at work; nevertheless, they are not often properly addressed by the organizations. In this paper we describe an application of the Delphi method to understand how externalities can be included in Occupational Safety and Health.

Keywords: Occupational Safety and Health; Externalities, Economic perspective, Delphi Method.

Resumen
Las organizaciones deben realizar periódicamente una evaluación de sus riesgos laborales con el fin de diseñar y poner en práctica medidas preventivas que sean necesarias y suficientes para mantener e l nivel de riesgo, los costos de la prevención y la seguridad en los niveles considerados aceptables por la organización. Por otra parte, la selección de las medidas a aplicar en una organización debe tener en cuenta tanto los costes internos como externos. Las externalidades son de gran importancia en términos de los costos de los accidentes de trabajo; sin embargo, a menudo no se tratan adecuadamente por las organizaciones. En este artículo se describe una aplicación del método Delphi para entender cómo los factores externos pueden ser incluidos en la Seguridad y Salud Laboral.

Palabras clave: Seguridad y Salud Ocupacional; Externalidades, Economía, Método Delphi.

1. Introduction

Both individuals and society lose emotionally and financially from injuries and accidents. Individuals can be understood to make their own best safety decisions if they have good information and the correct incentives. In this context, society may subsidize risk taking in many ways, thereby discouraging safety. Thus, regulation is required to deal with such externalities [2].

According to Agénor and Dinh [3], as defined by Putnam in 1993, there is a "social capital" that consists of "those features of social organization, such as networks of individuals or households, and the associated norms and values that create externalities for the community as a whole". Although a number of economists initially questioned the validity of classifying social interactions as a form of capital, an increasing number of them now acknowledge that social capital shares at least some similarities with physical and human capital in its inter-temporal dimension and its ability to generate external effects and future benefits. These externalities include information sharing among individuals and firms; and the matching of people to economic opportunities, mutual aid and insurance, which may affect expectations and individual behavior, as well as effective collective action. Social capital also enables agents to cope with market imperfections or imperfect institutions.

According to Varian [4], the definition of externality is that the action of an agent affects the living conditions of another agent not involved in that action. Externalities can also be defined as: "the uncompensated impact of actions of one person over the welfare of a spectator" [5]. The focus on human welfare, primarily used as a synonym for human utility, is due to the traditional utilitarianism of economics.

Samuelson and Nordhaus [6] state that externalities occur when companies or individuals impose costs or benefits on others who are not involved in the market. In the same way, Van Beukering et al. [7] consider that an externality occurs when an economic decision has an impact on the welfare of another economic agent not directly involved in the process. This results from the fact that the possibility of impact has not been properly addressed or has been disregarded. In general, an externality is present when the welfare function (Y) of some economic agent (utility or profit) includes real variables whose values are chosen directly by others (X), without special attention paid to the effect on the welfare of agent Y. Usually, externalities generated are only taken into consideration when a project needs or deserves an evaluation by a public entity. According to Cullis and Jones [8], externalities consist of social costs or benefits that manifest themselves beyond the realm of the project and influence the welfare of third parties without any monetary compensation.

The evaluation of projects of a private nature does not consider the effects on third parties arising from associated externalities. Indeed, the externalities generated by projects are in many cases difficult to quantify. This is the case, for example, with calculations that relate to the "value" of human life.

For Rebitzer et al. [9], externalities can be divided into the internalized externalities and the non-internalized externalities. According to Mann and Wüstemann [10], there are economic textbooks that suggest that externalities must be internalized in order to achieve a situation that is Pareto optimum.

Externalities are also important in the domain of OHS for both public and private organizations. Indeed, governments are usually concerned about the effect of negative externalities, not only in relation to the environment, which is the most typical external impact, but also to the area of occupational safety.

According to Dorman [1], broadly speaking, there are three general ways in which economics can be useful for OSH. Firstly, identifying and measuring the economic costs of occupational injury and disease can motivate governments to take these problems more seriously. This is true at all levels; the enterprise may be only dimly aware of the toll that worker ill-health takes on its performance, and national governments may not realize the impact of OSH problems on economic growth and development. Secondly, understanding the connections between the way firms and markets function and the types of OSH problems that arise are crucial for the success of public policies. Finally, the author considers that the protection of worker health and wellbeing is not the only objective of OHS in a modern society. Economic analysis can help to show when safeguarding working conditions are also complementary to other social goals, and it can enlighten any trade-offs, if indeed there are any.

Furthermore, when an organization performs a risk analysis that is integrated into the assessment of its OSH management system, several steps are suggested in order to solve the problems that are identified. Usually, the organization makes a detailed analysis of the monetary impact (positive or negative) for the organization in terms of each of the considered measures. However, it is also important to perform an analysis of the impact that those measures have on society, i.e., to measure the associated externalities. Indeed, as was previously explained, the measures taken by an organization in risk prevention may have an indirect positive effect (positive externality) on society, while no action, due to the costs for the organization, may have significant negative effect for society (negative externality). It follows that these effects should be duly considered in the decision-making process [11].

Furthermore, the quest for economic sustainability of OSH is acquiring greater visibility and strategic weight in corporate management. Thus, the process of calculating or estimating the economic value of OSH is a very relevant topic that needs to be analyzed in greater depth, as confirmed by Cagno et al. [14].

It is clear that different approaches, strategies and policies must be made according to the different consequences of accidents as well as which perspective is considered. It can also be concluded that the role of the workers in the prevention of accidents is essential, especially for the accidents that involve major injuries or death.

Thus, for all of these goals, a central issue is that of costs. On the one side, we have the costs of improving working conditions in order to reduce the incidence of injury and disease. On the other, we have the costs of not reducing the level of accidents; these are both tangible and intangible costs. For example, suppose, as a result of a catastrophic industrial accident, a firm loses half of its market share. This constitutes an enormous private cost to the firm. However, if sales are t still made by other firms, not all this private cost will be translated in a social cost. If the firm suffering the accident were more efficient than its competitors then the increase in the cost to society of supplying the goods would qualify as social. Second, not all social costs appear as private costs. For instance, a significant portion of the medical cost of occupational injury and disease in industrialized countries is indemnified by social insurance systems [1].

One of the priorities for research related to OSH in Europe during the period 2013-2020, undertaken by the European Agency for Safety and Health at Work [15], is to strengthen the research on the economic dimension of OSH, including estimating the socioeconomic costs of the consequences of poor or no OSH, and an analysis of costs and benefits of OSH prevention in order to support evidence-based policies and decision-making on society and enterprise levels.

The purpose of this paper is to contribute to the economic analysis that should be made in order to increase our understanding and management of OSH, with special emphasis on externalities resulting from work accidents.

2. Materials and methods

2.1. Delphi Methodology

The Delphi method is a particularly suitable research technique when there is an incomplete understanding of the subject under consideration: as is the case for externalities in occupational safety [16,17]. The method was developed by [18] of RAND Corporation in 1950 for a project sponsored by the U.S. Army. It was created as part of a post-war movement concerned with the prediction of possible effects of technological development in economic and social regeneration. The objective of the original study was to "obtain the most reliable consensus of opinion by a group of experts using a series of intensive questionnaires interspersed with controlled opinion feedback" [19].

This method is defined as a structured, interactive group communication and judgmental forecasting process which has the purpose of facilitating a systematic exchange of informed opinions among a panel of experts in order to develop a consensual understanding on a topic [19,20]. This is particularly the case in situations that are characterized by uncertainty, i.e., when objective, fact-based quantitative information is scarce or not reliable. It this context the Delphi method has proven to be effective [19].

Thus, the Delphi methodology is an exploratory study that allows the views of a panel of experts -which is called the Delphi panel- to be gathered. This process is carried out by conducting a series of questionnaires in, typically two or three rounds, on the subject under study [17]. In this research method, the results depend strongly on the quality of the questionnaire and the selection of experts [21].

The dimensions of heterogeneity for the purposes of a Delphi survey are manifold. For instance, individuals can differ in their age, gender, cultural and educational background, knowledge base, profession, values, attitudes, or tenure [22]. Furthermore, the Delphi methodology is characterized by the anonymity of the participants, the statistical representation of the distribution of results and the use of the feedback from the group to review the answers in a later round.

In this research method, the results depend strongly on the quality of the questionnaire and the selection of experts [16].

Although the method foresees several successive rounds of questionnaires, it can often be limited to two rounds without affecting the quality of the results, as has already been demonstrated in many studies [23,24].

According to Geist [25], the Delphi method can be used to determine important issues and also be used as a precursor or a first approach to the development of research.

Nevertheless, it is also important to be aware of its limitations, namely it is a laborious and time-consuming technique and it may be characterized by poor internal consistency and reliability of opinions among experts. This can lead to low reproducibility of forecasts based on the results obtained. Sensible results are obtained with respect to the ambiguity and reactivity of respondents; however, it is difficult to assess the participating experts' degree of knowledge.

2.2. Delphi Panel

For this study, a panel of experts specialized in the area of OSH has been chosen. Initially, 29 experts, including 13 academic experts, 8 technical/professional experts and 8 experts in consulting/audit have been contacted.

The questionnaire has been developed for the purpose of being applied to a panel of experts with different backgrounds. It was expected that with three rounds it would be possible to obtain important conclusions and to have a better understanding of the importance of the externalities in occupational safety [17].

2.3. Delphi Questionnaire

The questionnaire has a total of 11 questions related to externalities. For each question, the expert was asked to choose the answer on a scale of between 1 and 5 (1=very low, 2=low, 3=medium, 4=high, 5=very high); the possibility of having "no opinion" was also permitted. The variables studied are discrete, categorical and qualitative of ordinal type. The detailed structure of the questionnaire was presented in a previous publication [17]. The questionnaire was previously validated with three experts before being sent to all members of the expert panel. The questions are presented in Table 1 (section Results and Discussion).

The questionnaire was prepared as a PDF file in Adobe Acrobat and was then sent by email to each expert. After filling in the questionnaire, the expert just had to click on the "Submit Form" button and then the questionnaire was immediately transmitted electronically. This platform is user-friendly and allows the direct uploading of the answers in an Excel file for statistical analysis.

2.4. Methodology for the analysis of the answers

The questionnaires were used to reduce the "interquartile interval" (IQR), a measure of the deviation of the opinion of an expert from the opinion of the whole panel (median). The aim of the first questionnaire was then to calculate this deviation. If one or more rounds are performed, a higher consensus is to be expected on each issue [16,22,23]. According to Skulmoski et al. [26], the process can be considered as being concluded when the answers are near the consensus, according to appropriate statistical methods.

Quantifying the degree of consensus among Delphi panelists is an important component of Delphi data analysis and interpretation [27], but reaching consensus (as measured by a certain pre-defined threshold value) is not the general aim of a Delphi survey [28,29]. Among others, the IQR is often used as a measure for consensus in Delphi literature due to its robustness as a statistical measure [29,30]. The IQR is the range in which the middle 50% of the provided evaluations are located [31]. The inter-quartile range is the difference between the 3rd and 1st quartile in which 50% of core values lie. The IQR and the presentation of the quartiles allow an assessment of the degree of convergence of the answers. The quartiles can be used to help measuring the variability or dispersion of the observed data. The first quartile is a variable value such that the number of observations for lower values is 25% and upper 75%, i.e., the first inter-quartile (Q1) refers to 25% agreement between the experts, and the third quartile (Q3) refers to 75% agreement [32,33]. According to Bryman and Cramer [34], this measurement is more robust and less sensitive to isolated cases. It is also applicable in the case of ordinal variables, which are contained in the applied questionnaire.

There are many criteria to establish the moment the experts reached a consensus [35]. Indeed, depending on the scales used, different thresholds for the IQR can be defined to indicate that consensus among the experts has been achieved [28,32]. In the current study, the adopted criterion of consensus was when an IQR≤1 was achieved.

For all the 11 questions, the following statistical parameters have been calculated: mean, median, standard deviation and inter-quartile range.

The median is a better measure of the degree of group support for each factor; if it is high, we can conclude that there is a high level of support from the group. The standard deviation permits us to see the dispersion of results, which is directly related to the IQR.

The questionnaire sent to the experts for the second Delphi round included the treated results from the first round. In this second round, the "zone of agreement" was indicated with a red rectangle (see Fig. 1), considering the median value of the responses of the panel with a deviation of plus or minus one level. The answer given by the corresponding expert in the previous round has also been presented by using a red dot (see Fig. 1).

Fig. 1 shows an example of an extract from a second round questionnaire sent to the experts.

The experts have been invited to indicate their agreement or disagreement with respect to the median answer. The following alternatives were possible for each expert:

1. keep the original answer,
2. re-evaluate the initial answer and change it.

In the third round questionnaire, the experts' comments have also been included in order to help respondents to eventually re-evaluate the previous answer and change it. This helps to improve the level of consensus.

The statistical analysis of diagrams of extremes and quartiles in the first and third round was undertaken by using the statistical software IBM SPSS v.20.

3. Results and Discussion

3.1. Results of the first Delphi round

In the first round, we had the effective participation of a total of 20 experts from the 29 contacted. The results showed a good agreement (IQR≤1) in 8 of the 11 questions.

Fig. 2 presents the diagram of extreme quartiles and first round "Externalities".

The circle and the asterisk in Fig. 2 represent extreme cases. The circle represents the situation in which the minimum or maximum value is lower or higher than 1.5 IQR but less than 3 IQR (case considered as "moderate outlier"). The asterisk represents the extreme cases, which are significantly higher than 3 IQR (called "faroutlier" or "extreme outlier"). This notation follows the standard set in SPSS, as well as other conventional statistics programs (Statistica, SAS, R, etc.).

The second Delphi round was carried out in order to increase the consensus. In this round, there were 9 questions with IQR≤1 (including one with IQR = 0). The diagram of extremes and quartiles is very similar to the first round.

3.3. Results from the third Delphi round

In the third round, the experts were asked to reassess their responses, taking into account the results and also the comments made in the second round. In this round, the number of issues with IQR≤1 increased to 10.

The answers obtained from the panel in the third round are shown in Fig. 3. All the questions obtained a median of at least 4.0 (full line of Fig. 3).

Table 1 presents the statistical treatment of the results from the third (and final) round. For each question the following statistical parameters are presented:

• W: median. Positional statistical parameter: central value of those observed:
• m: mean. Positional statistical parameter: the sum of data values divided by the number of observations;
• s: standard deviation. Dispersion parameter: square root of the mean of the squares of deviations with respect to its median;
]]> IQR: interquartile interval. Dispersion parameter: difference between third and first quartiles.

3.4. Global discussion of the results

Table 2 presents the evolution of the panel's opinion throughout the three rounds.

The number of experts has reduced as the rounds, have gone on, which is normal in Delphi studies and does not invalidate the results. In fact, according to Okoli & Pawlowski [16], 14 is a good number of experts in Delphi studies; these authors suggest between 10 and 18 experts.

After round 3, the expert panel found that questions 1 and 9 in Table 1 were extremely important (median equals to 4.5 or 5) with IQR=1. Question 8 obtained the highest consensus (IQR = 0), with a median of 4.0.

The expert panel recognized the relevance of the implications for family stability, the direct calculation of the externalities based on the damages caused and the reduction of negative externalities through public solutions (taxes, fines, legislation, etc.). There was no consensus among the panel of experts on reducing negative externalities through private solutions in terms of the relationship between the company and the worker.

Translating risks and prevention measures in terms of the costs for companies employees' and for society may be used to support the reduction of negative externalities through public measures such as taxes, fines, legislation, etc. Responses showed a high consensus on the direct calculation of the externalities based on the damages caused.

The Delphi method is a suitable research technique when there is an incomplete or vague knowledge of the subject under consideration, as is the case of externalities in OSH.

This methodology allowed the assessment of selected experts to be collected through a questionnaire specifically designed for this purpose. It was then developed in three rounds, in order to increase consensus among the group of experts.

According to the issues highlighted by the expert panel, the externalities can be used to promote, support or to legislate on measures of preventing occupational hazards. Indeed, the incorporation of the effects of risks and preventive measures in terms of cost to the company for the employees and society may be used to support the reduction of negative externalities through public measures such as taxes, penalties and more restrictive laws.

The Delphi Method can be viewed as an important tool to gain a better understanding of the inclusion of externalities in occupational safety and health analysis, highlighting the more relevant aspects to be considered in economic studies in this particular field.

5. Acknowledgments

The authors would like to acknowledge all the experts who have participated in the Delphi panel. The authors also acknowledge the anonymous reviewers who helped to improve the overall quality of the current manuscript. This study was financially supported by FCT-Fundação para a Ciência e Tecnologia of Portugal, under the project ID/CEC/00319/2013.

References

[1] Dorman, P., The Economics of safety, health, and well-being at work: An overview. In Focus Program on SafeWork, International Labour Organisation. The Evergreen State CollegeTechnol, [On line]. 2000 [consulted February 2015]. Available at: http://www.oit.org/wcmsp5/groups/public/---ed_protect/---protrav/---safework/documents/publication/wcms_110382.pdf.         [ Links ]

[2] Lave, L.B., Injury as externality: An economic perspective of trauma. Accident Analysis & Prevention,19 (1), pp 29-37, 1987. DOI: 10.1016/0001-4575(87)90015-7        [ Links ]

[3] Agénor, P-R. and Dinh, H. T. Social capital, product imitation and growth with learning externalities. Journal of Development Economics 114, pp. 41-54, 2015. DOI: 10.1016/j.jdeveco.2014.11.008        [ Links ]

[4] Varian, H.R., Microeconomic Analysis. W.W. Norton & Company, Inc. 3rd Edition, 1992.         [ Links ]

[5] Mann, S. and Wüstemann, H., Multifunctionality and a new focus on externalities. The Journal of Socio-Economics, 37, pp. 293-307, 2008. DOI: 10.1016/j.socec.2006.12.031        [ Links ]

[6] Samuelson, P.A. and Nordhaus, W.D., Economics. McGraw Hill, 10th Edition, 1999.         [ Links ]

[7] van Beukering, P. van Drunen, M., Dornland, K. and Jansen, H., External economic benefits and costs in water and solid waste investments - Methodology, guidelines and case studies. Report number R98/11. IVM/EFTEC. ISBN 90-5383-632-2, 1998.         [ Links ]

[8] Cullis, J. and Jones, P., Public finance & Public choice: Analytical perspectives. Third edition. Oxford University Press. Oxford. UK, 2009.         [ Links ]

[9] Rebitzer, G., Ekvall, T., Frischknecht, R., Hunkeler, D., Norris, G., Rydberg, T., Schmidt, W.P. Suhh, S., Weidemai, B.P. and Pennington, D.W., Life Cycle Assessment - Part 1: Framework, Goal & Scope Definition, Inventory Analysis, and Applications. Environment International, 30(5), pp. 701-720, 2004. DOI: 10.1016/j.envint.2003.11.005        [ Links ]

[10] Mann, S. and Wüstemann, H., Multifunctionality and a new focus on externalities. The Journal of Socio-Economics 37, pp. 293-307, 2008. DOI: 10.1016/j.socec.2006.12.031        [ Links ]

[11] Ramos, D., Arezes, P. and Afonso, P., The role of costs, benefits and social impact of injuries and prevention measures on the design of occupational safety programs. In: Arezes et al. (eds), Occupational Safety and Hygiene, Taylor & Francis Group, London, 2013, ISBN 978-1-138-00047-6, pp. 153-157. DOI: 10.1201/b14391-32        [ Links ]

[12] HSE - Health and safety executive. Costs to Britain of workplace injuries and work-related ill health: 2010/11 update. HSE Books, UK, 2012.         [ Links ]

[13] ILO - International labour organization. Estimating the economic costs of occupational injuries and illnesses in developing countries: Essential Information for Decision-Makers. ISBN 978-92-2-127015-7. Geneva, 2012.         [ Links ]

[14] Cagno, E., Micheli, G., Masi, D. and Jacinto, C., Economic evaluation of OSH and its way to SMEs: A constructive review. Safety Science 53, pp. 134-152, 2013. DOI: 10.1016/j.ssci.2012.08.016        [ Links ]

[15] European Agency for Safety and Health at Work (EU-OSHA)., Priorities for occupational safety and health research in Europe: 2013-2020. Summary report. ISBN 978-92-9240-316-4, Luxembourg, 2014.         [ Links ]

[16] Okoli, C. and Pawlowski, S.D., The Delphi method as a research tool: An example, design considerations and applications. Information & Management 42, pp. 15-29, 2004. DOI: 10.1016/j.im.2003.11.002        [ Links ]

[17] Ramos, D., Arezes, P. and Afonso, P., Cost-benefit analysis in occupational health and safety: A proposal model. 11th International Probabilistic Safety Assessment and Management Conference and the Annual European Safety and Reliability Conference, PSAM11 ESREL 2012, 2, pp. 1273-1282.         [ Links ]

[18] Dalkey, N.C. and Helmer, O., An experimental application of the Delphi Method to the use of experts. Management Science, 9(3), pp. 458-468, 1963. DOI: 10.1287/mnsc.9.3.458        [ Links ]

[19] Linstone, H. and Turrof, M., The Delphi method, techniques and applications. Addison Wesley Publishing, 1975.         [ Links ]

[20] Donohoe, H.M. and Needham, R.D., Moving best practice forward: Delphi characteristics, advantages, potential problems, and solutions. International Journal of Tourism Research, 11(5), pp. 415-437, 2009. DOI: 10.1002/jtr.709        [ Links ]

[21] Förster, B. and von der Gracht, H., Assessing Delphi panel composition for strategic foresight - A comparison of panels based on company-internal and external participants. Technological Forecasting & Social Change 84, pp. 215-229, 2014. DOI: 10.1016/j.techfore.2013.07.012        [ Links ]

[22] Godet, M. Manual de Prospetiva Estratégica. Lisboa: Publicações Dom Quixote, 1993.         [ Links ]

[23] Landeta, J., El método Delphi. Barcelona: Ariel, 1999.         [ Links ]

[24] Vergara, S., Métodos de pesquisa em administração. São Paulo. Editora Atlas, 2005.         [ Links ]

[25] Geist, M.R., Using the Delphi method to engage stakeholders: A comparison of two studies. Evaluation and Program Planning 33, pp. 147-154. 2010. DOI: 10.1016/j.evalprogplan.2009.06.006        [ Links ]

[26] Skulmoski, J.G., Hartman, T.F. and Krahn, J., The Delphi method for graduate research. Journal of Information Technology Education, 6, pp. 1-21, 2007.         [ Links ]

[27] von der Gracht, H.A., Consensus measurement in Delphi studies: Review and implications for future quality assurance, Technol. Forecast. Soc. Chang. 79, pp. 1525-1536, 2012. DOI: 10.1016/j.techfore.2012.04.013        [ Links ]

[28] Linstone, H.A. and Turoff, M., Delphi: A brief look backward and forward, Technol. Forecast. Soc. Chang. 78, pp. 1712-1719, 2011. DOI: 10.1016/j.techfore.2010.09.011        [ Links ]

[29] Warth, J., von der Gracht, H.A. and Darkow, I.-L., A dissent-based approach for multi-stakeholder scenario development - the future of electric drive vehicles, Technol. Forecast. Soc. Chang. 80, pp. 566-583, 2013. DOI: 10.1016/j.techfore.2012.04.005        [ Links ]

[30] Ray, P.K. and Sahu, S., Productivity management in India: A Delphi study, Int. J. Oper. Prod. Manag. 10, pp. 25-51, 1990. DOI: 10.1108/01443579010005245        [ Links ]

[31] De Vet, E., Brug, J., De Nooijer, J., Dijkstra, A. and De Vries, N.K., Determinants of forward stage transitions: A Delphi study, Health Educ. Res. 20, pp. 195-205, 2005. DOI: 10.1093/her/cyg111        [ Links ]

[32] von der Gracht, H.A. and Darkow, I.-L., Scenarios for the logistics services industry: A Delphi-based analysis for 2025. International Journal of Production Economics 127, pp. 46-59, 2010. DOI: 10.1016/j.ijpe.2010.04.013        [ Links ]

[33] Astigarraga, E., El método Delphi. Universidad de Deusto, Facultad de CC.EE. y Empresariales. ESTE, [On line]. 2005, Available at: http://www.echalemojo.org/uploadsarchivos/metodo_delphi.pdf.         [ Links ]

[34] Bryman, A. e Cramer, D., Análise de dados em ciências sociais - Introdução às técnicas utilizando o SPSS. Celta Editora. Oeiras, 1993.         [ Links ]

[35] Fink, A., Kosecoff, J., Chassin, M. and Brook RH., Consensus methods: Characteristics and guidelines for use. Am J Public Health, 74(9), pp. 979-83, 1984. DOI: 10.2105/AJPH.74.9.979        [ Links ]

D. Ramos, has a PhD in Industrial and Systems Engineering from the University of Minho, Portugal. She is also Professor Honoris Causa of University of Tourism and Management in Skopje. Professor at Polytechnic Institute of Management and Technology (ISLA) and Polytechnic Institute of Cávado and Ave (IPCA). Member of Human Engineering research group - HErg - University of Minho. Member of the Research Center ALGORITMI, at the School of Engineering - University of Minho. Member of group of auditors of SGS Certification Body/ICS - General Society of Superintendence, SA. Health and Safety (OHSAS 18001) and (ISO 9001) area. ORCID: 0000-0003-2862-2883

P. Arezes, has a PhD. in Industrial and Systems Engineering from U.Minho, in Portugal, and is currently a full professor of Ergonomics and Human Factors at the same university. He is also a visiting fellow at MIT's AgeLab in the USA. He leads the Human Engineering research group and he is also the coordinator of Engineering Design and Advanced Manufacturing (EDAM) for the MIT Portugal Program at UMinhoand the chair of the steering board of the PhD program "Leaders for Technical Industries (LTI)" at UMinho. ORCID: 0000-0001-9421-9123

P. Afonso, holds a Diploma in Economics from the ISEG (Economics and Business School) - Technical University of Lisbon (UTL), a MSc. in Industrial Engineering from University of Minho, Portugal and a PhD. in Accounting and Finance from University of Manchester (UK). He is currently an auxiliary professor in the Production and Systems Department at University of Minho (Portugal) and author or co-author of more than 70 articles in international peer-reviewed conferences, books and journals. He is a researcher in Center Algoritmi (CAlg). He has also been a visiting scholar in several universities. ORCID: 0000-0003-3882-2491