ARTÍCULO ORIGINAL

Performance assessment of internal logistics for service companies

Evaluación del desempeño de logística interna para empresas de servicios

Vanessa Teresinha Alves, Julio Cezar Mairesse-Siluk, Alvaro Luiz Neuenfeldt-Júnior, Marlon Soliman*, Lissandro Dorneles Dalla-Nora

Núcleo de Inovação e Competitividade, Universidade Federal de Santa Maria. Av. Roraima, 1000, bulding 7, room 300. C.P. 97105-900. Santa Maria, Brasil.

* Corresponding author: Marlon Soliman, e-mail: marlonsoliman@gmail.com

(Received September 13, 2013; accepted November 25, 2014)

Abstract

The present research aims to propose a model for evaluating the performance of the internal logistics for a service company, based on Multicriteria Constructivist methods, towards a systematic mapping of the current context situation, aiming to find opportunities to improve its processes. It was found that, considering the difficulties of managing logistics flows, the method allows assisting in the identification, organization, measurement and integration of the variables through a systemic and participatory manner. As practical implications, it was provided conditions for the company to measure its actions, sustaining indexes for goal setting and periodic assessments. This research also presents the importance of this type of methodology for the provision of logistics services, a subject underexplored in the scientific literature, especially when using constructivist approaches.

Keywords: Internal logistics, performance measurement, organizational diagnosis, services sector, multicriteria constructivist methods

Resumen

La presente investigación tiene como objetivo proponer un modelo para evaluar el desempeño de la logística interna de una empresa de servicios, sobre la base de métodos multicriterio constructivistas, hacia un mapeo sistemático de la situación actual contexto, con el objetivo de encontrar oportunidades para mejorar sus procesos. Se encontró que, teniendo en cuenta las dificultades de la gestión de los flujos logísticos, el método permite que ayudan en la identificación, organización, medición e integración de las variables a través de una manera sistémica y participativa. Como implicaciones prácticas, se dispuso condiciones para la empresa para medir sus acciones, el mantenimiento de los índices para el establecimiento de metas y evaluaciones periódicas. Esta investigación también presenta la importancia de este tipo de metodología para la prestación de servicios de logística, un tema poco explorado en la literatura científica, especialmente cuando se utilizan enfoques constructivistas.

Palabras clave: Logística interna, medición del desempeño, diagnóstico organizacional, sector de servicios, métodos multicriterio constructivistas

Introduction

The opening of the economy, its consolidation and stronger business presence with global reach led to the change from the old style of business management based on productivity, for a new model based on competitiveness [1, 2]. This modern concept employed nowadays involves the characteristics of intangibility, heterogeneity, inseparability, perishability and assumed major proportions, due to the appearance of new processes for creating and conducting business. Grounded mainly in information technology, these new approaches assume that resources rationalization, with the consequent value addition and costs reduction, has become a vital point for the development of enterprises and nations [3, 4].

Noted the complexity, it is imperative to consider different points of view for the generation of competitiveness, with an eye on the past as a way to strengthen the arrangements, providing security on market inconstancy.

Thus, the main problems of the service sector, in relation to logistics approach, are divided into two properties: storage impossibility and incommensurability. The first implies that traditional strategies for manufacture management and demand forecast may not be suitable in this environment, since services can not be stored or transported. The second reflects the measures use difficulties, mainly to assess the services operations, once it is consumed as soon as produced [4, 5].

In contemporary organizational concept, logistics appears with a strategic facet, not only because of materials management and physical distribution, but also for providing values of time and place for customers, to become an element that distinguishes for organizations, with agility, flexibility and integration of its internal and external channels [6-8].

[9] describe that the logistics concept can be separated into three basic points: supply (suppliers), plant (internal) and distribution (customers), with the aggregation of other various sub-activities, which represent a group that is often defined as highly empirical, which results in negative effects that directly influence the outcome and final organizations performance, necessarily requiring tools to monitor its situation.

Therefore, the present research aims to propose a model for evaluating the performance of the internal logistics for a service company. The design is justified, among other reasons, by checking the growth and decline of the classical division between the industry and services sectors. The importance, under the social point of view, is justified by the statistics, where it is noted that the last decades have been marked by the dynamism of service sector. In numerous countries, this resulted in growth rates higher than the overall economy [10, 11].

From the organizational perspective, the search for competitiveness has required of the organizations concerns about its performance systems, in a manner to search for fast solutions, accurately and efficiently [12-15], in manner that this is one of the most important investigation gaps explored during the research.

Methodology

To achieve the proposed objective, the research was decomposed into three steps: a literature review, the development, and the practical application of modeling, as shown in figure 1.

The first step involved studies that supported the understanding of the context, in order to study concepts of the service sector and logistics, in addition to the state of the art regarding performance measurement through Multicriteria Constructivist methods. This procedure consisted of a literature review of secondary sources, based on books and scientific articles, being developed with the support of publishers as Scientific Direct®, Emerald® and Scopus®.

The development is understood by compliance with the guidelines assumed by [16-20] regarding the concepts of Multicriteria Decision Aid (MCDA), which was chosen based on the verification flow described by [21]. The fulfillment of this flow resulted in the construction of the measurement modeling proposed for evaluating the performance of the internal logistics, supported with the variables identified mainly in researches by [2, 6, 8, 11, 22, 23].

The application of the modeling was realized in the company named Press®, located in Brazil, which operates mainly with services for companies in the telecommunications industry, having contract with the largest existing on the national scene.

Modeling development

Considering this context, the Multicriteria Decision Aid (MCDA) are customarily recognized as tools to support the organizational performance measurement, requiring primarily the definition of criteria in a flexible manner to its constitution, having the ability to add features considered important for the purpose of providing transparency and systematization of the process related to the problems [19, 24].

In accordance with what has been established on the methodological step, the basis is designed through the fulfillment of the three steps shown by figure 2, where the structuring phase, according [25], seeks to understand aspects of information presentation and the way that it might cause a significant impact on decision making [16, 26].

Based on the methodology proposed in the literature, it is considered that the main factors influencing the system performance can be considered as variables related to people logistics, information, or materials and equipment. For people logistics, [23] refer that any and all organization depends on the human performance to its success, in a greater or lesser degree, using their knowledge and putting it into action quickly and effectively for searching satisfying solutions [22, 27].

In the literature studied, it has been identified the logistics management of information as a strategic resource vital to the process of decision making [28-30]. Companies have in the information flow the link that unites the components coordinates, which facilitates the maintenance of balance and integration in an environment of increasing mutation [31, 32].

[2] states that knowledge concerning management is not coded and systematically developed in the same way. Although there are extensive research in information knowledge, its characteristics and alternatives to manage tends to be fragmented, misunderstood and misapplied. The authors confirm that its continuous improvement generates substantial advantage to the market and competition. [28], in turn, believe that the strategic process it is a structured set of activities that include the way companies obtain, distribute and use knowledge.

[29] clarify the objectives that involve such as: planning information policies, developing and maintaining systems and services, promoting organizational efficiency; meeting the demands for information coming from inside and outside [30, 33].

Regarding the logistics flow of materials and equipment, the explanation is intertwined with the concept of logistics, according to [6, 8, 34], it is responsible for decreasing the gap between production and demand, wich needs an effective management from purchasing all necessary items for the execution, to prevent any loss due to lack of inventory control.

Basically, the production activities require the handling of three basic elements: man, material and equipment; so that material movement, in whatever process, generates a flow of information to coordinate it, through an integrated sequence of tasks [6, 8].

Regarding the definition of the actors involved in the decision process, it is necessary to identiffy the entities that have an interest in the results, in order to be designated as: decision makers (have decision-making power), facilitator (responsible for promoting and supporting the process through tools built for this purpose) and acted (suffer the consequences of decision making, participating indirectly of the system) [17, 35].

Set the context and the actors, it goes to the model construction, which is presented into three sub-steps, depending on the assumptions described by [17]. The first, concerns the construction of the Primary Elements of Assessment (PEA) and its concepts, objectives, goals, values, as well as actions, options and alternatives. This was realized through the application of the diagnosis tool developed by the Innovation and Competitiveness Group (NIC) of the Federal University of Santa Maria. The responses related to each constituent elements of a GUT matrix (Gravity, Urgency, Tendency) were used to assess which are most representative for the the system.

In a second step, it is structured the map of relationships formed by middle-concerns (goals with less subjective and more specific focusing) and end-concerns (strategic, subjective and broad objectives), related by links of influence. To build the map, the decision maker should answer about the means to be used to achieve a concept and the purpose for which it is intended [35].

The version consists of the upper part, which are the Fundamental Points of View (FPV), and the lower part, which presents its decomposition, called by Elementary Points of View (EPV). The FPV represent the values and goals expressed by most relevant decision makers within the process and serve as guides for defining the characteristics and evaluation of the actions that are of interest. The EPV explain each one, and from this decomposition comes the ''Hierarchical Structure of Value'' [17, 36].

Based on information contained in the maps of means-ends relationships, it comes the last step of the model structure, the descriptors, allowing the identification of an impact levels, ordered in terms of preference, according to the decision maker's value system. Therefore, one can not say that there is an ''optimum'' descriptor, but rather one that best represents the measurement of certain aspect [35].

The descriptors refer to the way to assess the potential actions according to its values, so it is necessary to identify the type of scale to be used as well as the method for building the function value, which is used to sort the intensity preferably between levels of impact or potential actions. The reference levels are identified according to the levels namely ''Neutral'' and ''Good''. Other levels that impact between these two reference levels represent a situation with a performance that, at least, meets the expectations of decision makers (competitive) [17].

After the determination of reference levels, it is necessary to define the scales of impact make impact levels equivalent throughout the model and thus make a comparison between them. Therefore, the decision maker and facilitator proceed reading all descriptors and define values for the levels ''Neutral'' (zero) and ''Good'' (hundred) and consequently for the other scales [17, 37].

According to the structured flow in figure 2, in the evaluation phase occurs the measurement of the impact of actions through the value judgments of those involved, which allows the facilitator to provide resources and recommendations for decision makers, being achieved through the formulation of value function, the determination of substitution rates and the global assessment of those actions where value functions are constructed for each descriptor. For this, they express their preferences of one over another, and in the end, there is the predilection expressed numerically.

In order to proceed with the evaluation, it was decided to use the Multicriteria Value Theory (MVT), based on multicriteria optimization, fitting the construction of the functions that mathematically represent human decisions and seek to provide an analytical description of the systems [19]. If an attribute is considered unimportant in front of the other attributes, it will receive a weight (assigned value) lesser than the weight given to those of greatest importance [38].

The information provided by the diagnosis will generate, through the ''Hierarchical Structure of Value'', substitution rates, which according to [17] demonstrate the performance loss that a particular criterion must suffer to offset the performance gain in another, used for such interaction the technique called Swing Weights, which consists in balancing the weights for each area according to the judgment made by the decision maker for EVP determined in the model [19].

The decision maker considers each descriptor separately, assigning its relative use. The weights of the descriptors are then measured in order to express its importance. The value for each alternative utility is obtained by multiplying the value of each weight by its respective descriptor in each of the alternatives. Calculating the sum of the individual impact of this factor to each descriptor is obtained in total utility each alternative [39].

In the next sub step, performance evaluation, it is calculated the performance obtained by each criterion. This is done through the use of an additive aggregation function as a weighted sum, using the substitutions rates defined [40]. Therefore, the overall value function is given by equation (1) to the ''A'' alternatives proposed for the problem,

where is the partial value, w_i are substitution rates and i is the index for the selected descriptors, being proposed for this model the use of Visual Interactive Sensitivity Analysis® (VISA) and Microsoft Excel® softwares.

The last phase deals with compilation of the finished model to be submitted to the organizations, so as to allow, in the end, the suggestion of potential actions that tend to enhance future situation regarding the current one. This procedure is achieved by means of a semistructured diagnostic that contains a question for each EPV predetermined, seeking to identify the criteria which the company is below expectations and those that have a higher potential global contribution [17, 36].

Results and discussions

Under the proposed evaluation framework, as initial activity it is necessary to identify the actors who participate in this study. This research was carried out with three decision makers: The Owner is a civil engineering and is the responsible for the strategic planning and the coordination of all operational areas since 2009. The general manager has a degree in business administration and is responsible since 2009 for supporting operational areas with accounting.

Finally, the operations manager holds an electrical engineering degree and acts coordinating the processes to meet strategic goals since 2010. The authors, with previous experiences in logistics and MCDA, were listed as the facilitators and the company's employees as the acted ones, once they do not participate in the decision process.

Therefore, the proposed diagnosis is admixed by the establishment of three FPV, as described above, being FPV 1 related to people logistics, FPV 2 to information and FPV 3 to equipment and materials, in which the first two were considered as strategic nature and the third as operational.

The next action is related to the binding of each of the 78 issues existing in the NIC diagnosis according to the defined FPV, through GUT matrix, so as to be able to define the Primary Elements of Assessment (PEA). Therefore, the allocation of responses corresponding to the tendency was the Owner's responsibility, gravity to the General Manager and urgency to the Operations Manager, where a total of 43 PEA relevant to the context were identified, later named as EPV, being seventeen related to FPV 1, seventeen to FPV 2 and nine to FPV 3, described in detail in table 1, allowing to proceed with the development of a hierarchical structure of values or also called decision tree.

Thus, for each EPV were defined ordinal scales that best represent its judgment value, thus establishing reference levels that determine the ranges of limits to the level considered good for benchmarking, and the neutral level considered compromising, being the interval between these representative of a competitive situation.

In the evaluation phase it is associated the substitution rates for every FPV and EPV, which express, within the judgment of the decision makers, the loss of performance of a criterion over another. To determine the rates mentioned, it was used as a support the VISA software, where the method applied to determine the rates was the Swing Weights. This uses levels to weight distribution, so that the scores will be between the minimum and maximum value, determined by a value function constructed for that specific sub-criteria, and in which decision makers should assess the impetus of a change values for the interval.

The definition of the weights was carried out by first assigning the weights to the EPV and later for FPV, being possible to consider that the strategic and operational processes showed very distinct results, simultaneously reaching 78% and 22% of the total, an aspect that relates the position of the market in which the company operates. Observing the variables, the FPV 1 represents 49.5% and FPV 2 refers to 50.5%, in relation to the strategic process, while operational PV 3 represents the entire value for this group.

After defining the ways in which alternatives will be evaluated, through the descriptors construction, scales and substitution rates, it is obtained the model to evaluate the logistics performance, aiming to refer to the status in which the company Press® is, being presented at the end an index of overall performance I(A) 70 out of the 100 points attainable. This index refers to the level of the enterprise market competitiveness, far from the compromising level, but with the possibility of improvement to achieve what we defined as level of excellence.

In evaluating the EPV, in FPV 1, which achieved a I(FPV 1)=70, five (1, 4, 9, 11 and 12) reached the level of excellence, five (3, 5, 7, 14 and 15) are close to the excellence and other seven (2, 6, 8, 10, 13, 16 and 17) had rates below the midline. FPV 2 reached I(FPV 2)=69 and presented eight EPV that necessity some performance improvement (8, 10, 11, 12, 13, 14, 15, 16) and, finally, the FPV 3 reached I(FPV 3)=86, in which eight of the nine EPVs reached an outcome considered satisfactory, with only one exception related to EPV 3.8 as visible of gaining competitiveness. Given the results presented and the total score FPV 3 comes as the group with the highest rate, thereby prioritizing improvement actions focused in the other two groups.

As noted in the analysis of results, the way the descriptors are constructed allows stimulating creativity in the process of proposing actions to improve performance. The recommendations are suggested getting the discretion of decision makers for the choice and implementation. The improvements were applied to the sub-criterias with the lowest performance reported within the whole set analyzed, as shown in table 2.

Besides the suggested strategy, it is also presented the actions for each of the points of view, with goals to be achieved to be able to change status and improves the overall performance of the company. Faced with the prospect of such improvements, figure 3 shows the new rates achieved through improvement goals, making clear that if applied to the company, performance can be enhanced.

After defining the actions, the model recommends its monitoring regarding the care of these actions through a schedule and its effectiveness in the points of view performance, since there are still further possibilities for improvement in the evaluated points, achieving the goal of the full level of excellence in logistics.

Conclusions

Considering the difficulties of managing logistics flows, the method developed allows assisting in identification, organization, measurement and integration of the variables in a systemic manner, since it considered the needs, aspirations and perspectives of the authors. As a distinct factor, the study shows the importance of measures related to internal logistics in providing services as being an important factor, although little substantiated in the literature.

At the end, it was possible to present local and global performances of the variables studied, being evaluated regarding their continuity, according to the aspects defined by the decision makers. With this intention suggestions for improvements to the elements identified as compromising were presented, seeking to contribute to the overall evaluation indexes to reach a level close to excellence. Given the suggested recommendations, it can be noticed that the information logistics has better perfomance.

This analysis also examined the proximity between the operational focus of logistics actions, being concerned with the strategic factor as well as the operational one, which proves to be an intelligent and rational approach to get the best results working with a system coordinated and adjusted to its main objective, the total integration of logistics flows that compose it.

A factor highlighted in the proposed methodology is the ability to produce knowledge and increase understanding of the decision-maker in relation to a particular decision context, based on the literature, in order to propose alternatives to improve the performance of companies, where the setting of goals for individual and collective performance will allow better management and control. The research unveils the possibility of future research involving organizations from different sectors enabling the analysis and comparison between them and the market. We also expect to apply the modeling in more companies, in order to visualize how this approach behaves in different cenarios, to observe the main managerial contributions that this tool could provide in different scenarios.

References

1.  P. Drucker. Management. Revised edition. Ed. Harper Business. New York, USA. 2008. pp. 608.         [ Links ]

2.  M. Porter. On Competition. Updated and expanded edition. Ed. Harvard Business Press. Watertown, USA. 2009. pp. 544.         [ Links ]

3.  C. Grönroos. Service Management and Marketing: Customer Management in Service Competition. 3^rded. Ed. Wiley. New York, USA. 2007. pp. 496.         [ Links ]

4.  J. Fitzsimmons, M. Fitzsimmons. Services Administration. 6^th ed. Ed. Bookman. Porto Alegre, Brazil. 2010. pp. 584.         [ Links ]

5.  I. Plumb, A. Zamfir. ''New Hypothesis on Service Management Within the Global World''. Theoretical and Applied Economics. Vol. 18. 2011. pp. 121-128.         [ Links ]

6.  R. Ballou. Business Logistics: Supply Chain Management. 4^th ed. Ed. Prentice Hall. Lebanon, USA. 2003. pp. 816.         [ Links ]

7.  J. Bastidas, J. Gómez, J. Cabrera. ''Modelo para la priorización dinámica de despachos de vehículos utilizando el processo analítico jerárquico''. Revista Facultad de Ingeniería Universidad de Antioquia. Vol. 48. 2009. pp. 201-215.         [ Links ]

8.  D. Bowersox, D. Closs, M. Cooper. Supply Chain Logistics Management. 4^thed. Ed. McGraw-Hill. Columbus, USA. 2012. pp. 496.         [ Links ]

9.  N. Slack, R. Chambers, R. Johnston, A. Betts. Operation and process management: principles and practice for strategic impact. 2^nd ed. Ed. Prentice Hall. Lebanon, USA. 2008. pp. 560.         [ Links ]

10.  D. Kalenatic, L. Méndez, K. Valbuena, L. Rodriguez. ''Metodología de planeación logística baseada en gestión de proyectos y dinámica de sistemas en empresas prestadoras de servicios''. Revista Facultad de Ingeniería Universidad de Antioquia. Vol. 58. 2011. pp. 208-218.         [ Links ]

11.    Concil of Logistics Management. Facts on the global supply chain. Available on: http://www.cscmp.org./media-center/facts-global-supply-chain. Accessed: July 4, 2013.         [ Links ]

12.  A. Gunasekaran, C. Patel, R. McGaughey. ''A framework for supply chain performance measurement''. International Journal of Production Economics. Vol. 87. 2004. pp. 333-347.         [ Links ]

13.  M. Sharma, R. Bhagwat. ''An integrated BSC-AHP approach for supply chain management evaluation''. Measuring Business Excellence. Vol. 11. 2007. pp. 57-68.         [ Links ]

14.  M. Shi, W. Yu. ''Supply chain management and financial performance: literature review and future directions''. International Journal of Operations & Production Management. Vol. 33. 2013. pp. 1283-1317.         [ Links ]

15.  P. Taticchi, F. Tonelli, R. Pasqualino. ''Performance measurement of sustainable supply chains: a literature review and a research agenda''. International Journal of Productivity and Performance Management. Vol. 62. 2013. pp. 782-804.         [ Links ]

16.  B. Roy. Multicriteria Methodology for Decision Aiding. Ed. Kluwer Academic Publishers. Dordrecht, Netherlands. 1996. pp. 293.         [ Links ]

17.  L. Ensslin, G. Montibeller, S. Noronha. Decision Support: Methodology for Structuring Problems and Multicriteria Evaluation of Alternatives. 1^st ed. Ed. Insular. Florianópolis, Brazil. 2001. pp. 296.         [ Links ]

18.  S. Hatefi, S. Torabi. ''The common weight MCDA-DEA approach to construct composite indicators''. Ecological Economics. Vol. 70. 2010. pp. 114-120.         [ Links ]

19.  C. Gomes, L. Gomes. Managerial decision making: Multicriteria Approach. 4^th ed. Ed. Atlas. São Paulo, Brazil. 2012. pp. 360.         [ Links ]

20.  P. Meyer, A. Olteanu. ''Formalizing and solving the problem of clustering in MCDA''. European Journal of Operational Research. Vol. 227. 2013. pp. 494-499.         [ Links ]

21.  A. Guitouni, J. Martel. ''Tentative guidelines to help choosing an Appropriate MCDA method''. European Journal of Operational Research. Vol. 109. 1998. pp. 501-521.         [ Links ]

22.  S. Hronec. Vital Signs. 1^st ed. Ed. Pearson. Upper Saddle River, USA. 1994. pp. 256.         [ Links ]

23.  C. Barlett, S. Ghoshal. Managing Across Borders: The Transnational Solution. 2^nded. Ed. Harvard Business Press. Watertown, USA. 2002. pp. 416.         [ Links ]

24.  T. Saaty, L. Vargas. Methods, concepts & applications of the Hierarchy Process. 2^nd ed. Ed. Springer. New York, USA. 2012. pp. 396.         [ Links ]

25.  M. Bazerman, D. Moore. Judgment in Managerial Decision Making. 6^th ed. Ed. Wiley. New York, USA. 2004. pp. 288.         [ Links ]

26.  R. Keeney, H. Raifa. Decision with Multiple Objectives, Preferences and Value Trade Offs. 1^st ed. Ed. Wiley. New York, USA. 1976. pp. 592.         [ Links ]

27.  J. Hiatt, T. Creasey. Change Management: The People Side of Change. 2^nd ed. Ed. Prosci Inc. Loveland, USA. 2012. pp. 148.         [ Links ]

28.  T. Davenport, J. Harris. Competing on Analytics: The New Science of Winning. 1^st ed. Ed. Harvard Business School Press. Watertown, USA. 2007. pp. 240.         [ Links ]

29.  M. Oliveira, M. Bertucci. ''The small and medium enterprise and information management''. Information and Society. Vol. 13. 2003. pp. 1-16.         [ Links ]

30.  A. Beal. Strategic information management. 1^sted. Ed. Atlas. São Paulo, Brazil. 2004. pp. 144.         [ Links ]

31.  G. Rummler, A. Brache. Improving Performance: How to Manage the White Space on the Organization Chart. 3^rd ed. Ed. Wiley. New York, USA. 2012. pp. 256.         [ Links ]

32.  S. Snell, G. Bohlander. Managing Human Resources. 16^th ed. Ed. Cengage Learning. Independence, USA. 2012. pp. 864.         [ Links ]

33.  K. Laudon, J. Laudon. Management Information Systems. 8^th ed. Ed. Prentice Hall. Lebanon, USA. 2004. pp. 480.         [ Links ]

34.  M. Christopher. Logistics and Supply Chain Management. 4^th ed. Ed. FT Press. Upper Saddle River, USA. 2011. pp. 288.         [ Links ]

35.  F. Rosa, S. Ensslin, L. Ensslin, R. Lunkes. ''Environmental disclosure management: a constructivist case''. Management Decision. Vol. 50. 2012. pp. 1117-1136.         [ Links ]

36.  R. Lacerda, L. Ensslin, S. Ensslin. ''The performance measurement framework in portfolio management: A constructivist case''. Management Decision. Vol. 49. 2011. pp. 648-668.         [ Links ]        [ Links ]

38.  M. Koksalan, J. Wallerius, S. Zionts. Multiple criteria decision making: from early history to the 21st century. 1^st ed. Ed. World Scientific Publishing Company. New York, USA. 2011. pp. 200.         [ Links ]

39.  L. Gomes, M. Araya, C. Carignano. Decision making in complex scenarios: Introduction to discrete methods of multicriteria decision. 1^st ed. Ed. Cengage Learning. São Paulo, Brazil. 2003. pp. 168.         [ Links ]

40.  R. Stolt, L. Ensslin. ''Avaliação de pilotos da Very Light Jets utilizando-se a MCDA-C''. Conexão SIPAER. Vol. 1. 2009. pp. 85-103.         [ Links ]