ARTÍCULO ORIGINAL

 

DOI: 10.17533/udea.redin.n77a12

 

A 360–degree process improvement approach based on multiple models

 

Un enfoque de mejora de procesos de 360 grados a partir de múltiples modelos

 

 

César Jesús Pardo–Calvache¹*, Félix Oscar García–Rubio², Mario Gerardo Piattini–Velthuis², Francisco José Pino–Correa¹, Maria Teresa Baldassarre³

¹Departamento de Sistemas Facultad de Ingeniería Electrónica y Telecomunicaciones, Universidad del Cauca. Calle 5 # 4 – 70. C. P. 190002. Popayán, Colombia

²Grupo de investigación IDIS, Facultad de Ingeniería Electrónica y Telecomunicaciones, Universidad del Cauca. Calle 5 # 4 – 70. C. P. 190002. Popayán, Colombia

³Grupo de investigación ALARCOS, Departamento de Tecnología y Sistemas de la Información , Universidad de Castilla–La Mancha. Paseo de la Universidad, 4. C. P. 13071. Ciudad Real, España.

⁴Department of Informatics, Software Engineering Research and Practices (SER&P), University of Bari. Vía E. Orabona 4. C. P. 70126. Bari, Italy.

* Corresponding author: César Jesús Pardo Calvache, e–mail: cpardo@unicauca.edu.co

ISSN  0120–6230

e–ISSN 2422–2844

 

(Received February 28, 2015; accepted August 24, 2015)

 

 

---

ABSTRACT

Several models and methodologies have been defined in order to support organisational process improvement. The implementation and institutionalisation of these approaches allow organisations to improve, mature, acquire and institutionalise best practices and management systems from multiple approaches. However, there are two issues, which have to be kept in mind. On one hand, it is possible to find several similarities amongst improvement, management and governance approaches. Experts and practitioners can thereby save, improve and optimize the organisational efforts using the best parts of existing models as building blocks; they can thus be prepared to deconstruct models, aiming for their designs to meet multiple needs. On the other hand, nonetheless, there are other factors which may influence, for example, compliance, or those aspects related to structural differences such as terminology, size, process, element structure, content, granularity, and complexity, which make difficult to work in multi–model environments. This being the case, the people involved need a map or guideline telling them how to carry out the harmonisation of models and standards that have to be implemented inside their organisations. In the quest to help support the work of harmonization of multiple models, this paper presents a framework that defines elements needed for the harmonization of multiple reference models to occur, as well as its application to three case studies. The results obtained show that the framework proposed has allowed the harmonization of several models.

Keywords: Harmonization of multiple models and standards, software process improvement (SPI), models, standards

---

RESUMEN

Diferentes modelos y metodologías se han definido para apoyar la mejora de procesos de las organizaciones. La implementación e institucionalización de estos enfoques permiten a las organizaciones mejorar, madurar, adquirir e institucionalizar las mejores prácticas y sistemas de gestión a partir de múltiples enfoques. Sin embargo, hay dos cuestiones que se deben tener en cuenta. Por un lado, es posible encontrar varias similitudes entre los enfoques de mejora, gestión y gobierno, de esta manera, expertos y profesionales pueden ahorrar, mejorar y optimizar los esfuerzos de la organización usando las mejores partes de los modelos existentes como bloques de construcción; en este sentido, tanto expertos como profesionales deben estar preparados para reconstruir los modelos con el objetivo de diseñar las prácticas de una organización de acuerdo a sus múltiples necesidades. Por otro lado, sin embargo, hay otros factores que pueden influir en el cumplimiento, lo que hace difícil para trabajar en entornos multimodelo, por ejemplo, los aspectos relacionados con las diferencias estructurales tales como la terminología, tamaño, estructura de elementos de proceso, contenido, granularidad y complejidad. En este sentido, las personas involucradas necesitan una ruta o guía que les permita saber cómo llevar a cabo la armonización de modelos y estándares que tienen que aplicar en sus organizaciones. Con el objetivo de facilitar la labor en la armonización de múltiples modelos, este artículo presenta un marco de trabajo que define los elementos necesarios para apoyar la armonización de múltiples modelos de referencia. Asimismo, se presenta su aplicación en tres estudios de caso. A partir de los resultados obtenidos ha sido posible armonizar múltiples modelos a través del marco de trabajo (framework) propuesto.

Palabras clave: Armonización de múltiples modelos y estándares, mejora de procesos de software (SPI), modelos, estándares

---

1. Introduction

Currently, there is a wide range of models that have been developed and which can be taken as a reference model (RM) to improve processes inside an organization. In 1999, for instance, Moore identified approximately 315 standards, guides, handbooks, and other prescriptive documents which were taken as RMs and maintained by 46 different organizations [1]. Nowadays, RMs provide best practices to cover different needs; e.g. Information Security Management System (ISMS) such as ISO 27001, Information Technology Governance Processes (IT Governance) and Services Management including ITIL, COBIT, ISO 20000, CMMI–SVC, or quality management systems like ISO 9001, EFQM, Six–Sigma, or even those in much more specific domains such as software development, maintenance, acquisitions –CMMI–DEV, CMMI–ACQ, ISO 90003, ISO 15504, ISO 12207–, and so forth.

Some models are widely used in the industry to improve organization competitiveness, while others are required as mandatory standards and become a regulation method in certain market niches. Organizations can benefit from this high number of models and standards when assessing and institutionalizing new or improved processes and, as a consequence, becoming more competitive and producing high–quality products [2]. Independently of the model to be used, its implementation requires specific experience and knowledge, along with a high degree of effort and investment, as key factors for its success. All this implies that the task is not easy and that there is a significant risk of failure [3].

One of the most important things about the huge amount and variety of models to select from, is that they can be applied to support multiple needs [4]; however, this proliferation can make that the organizations become confused about what the most suitable model for them is. In addition, there are other issues that need to be resolved; for instance, the way to reconcile the structural differences, size and terminology between multiple models. We cannot forget that each model has its own features, which can be reflected through its approach, processes structure, definitions, concepts, vocabulary, amongst other things. Although this scenario can be quite heterogeneous, it is possible to find some relationships between different models from some characteristics they have in common; e.g. models with similar approaches such as ISO models usually share similar quality objectives and, therefore, comparable practices [5]. Companies can benefit because implementing multi–model processes from shared quality goals reduces the costs of adopting multiple models [6].

However, not all relationships are easy to establish between all models. Furthermore, models defined are not always implemented by the same body at equal times inside a company. These dissimilar organizational points of view cause a problem regarding model compliance and standards to arise; e.g. structural differences between COBIT and ISO 9001 make difficult to establish their overlap. This disagreement causes difficulties in understanding them, together with compliance and unification issues in its adoption, which at the same time implies greater efforts, time consumption and associated costs as opposed to when only one model or standard is used for process improvement. Problems have also arisen concerning ambiguity, instability, subjectivity, incompatibility and transformability, as well as the benchmarking of process elements [7].

Currently, software organizations need guidance in identifying and resolving differences and similarities between multiple models susceptible of being implemented by them, in order to improve their processes. Therefore, in an attempt to offer a solution that facilitates the harmonization of multiple models, this paper presents a Harmonization Framework (HFramework) –a solution that provides a 360–degree approach to support the multi–model process improvement; i.e. when several and different models need to be implemented and institutionalized in a company–. The findings obtained from the application of this harmonization proposal in three case studies, show that it allows the use of different models when carrying out software process improvement in a systematic manner.

The paper proceeds as follows. In Section 2 an analysis of related work is presented. Section 3 illustrates an overview of HFramework which delimits a set of elements for defining suitable harmonizing strategies that support strategic business objectives by bringing into consonance the differences between multi–models. Section 4 shows the research methods applied in the case studies. Section 5 summarizes three case studies where HFramework was applied. Section 6 partially exemplifies a unified practice between ISO 9001 and CMMI, also illustrating how HFramework supports the integration of models. Section 7 presents the lessons learned, and Section 8, conclusions and upcoming future work.

2. Related works

Some early works provide interesting proposals that show a growing interest in recent years on the part of the software engineering community regarding process improvement environments where multiple models are involved. Figure 1a exposes the studies found and which have been organized in five periods of time (each one of five years), from 1990 to 2015. It is important to highlight that this analysis does not include the studies which can arise in 2015. In Figure 1a it is possible to notice that there is an increase in the number of studies published lately; i.e. many researchers are interested in this research field affecting the software industry. Part of this growing interest occurs due to the fact that the governments are paying more attention to software industry. As a result of this, it is possible to find laws with more benefits for this sector, one of them being process and practice improvement inside small and medium enterprises that currently occupy a representative place in worldwide software development.

Figure 1b shows the percentage of studies organized from the following features: (i) studies presenting a solution based on mappings in a unilateral direction, (ii) studies describing ontologies to represent the key elements of particular domains and (iii) studies providing a solution for supporting multi–environments; the latter is one of the groups which provides solutions to support the implementation of more than two models at the same time. Table 1 summarizes the studies classified from the features above mentioned.

In the light of the situation above described, the following sections propose a solution to support the harmonization of multiple models.

3. Supporting the multi–model process improvement with HFramework

HFramework was developed to provide the conceptual, methodological and technological support necessary for facilitating the harmonization of multiple models. Figure 2 shows the elements inside the HFramework.

3.1. Conceptual Framework

The conceptual framework provides the means necessary to understand the complexities in aligning multiple models. To this end, the conceptual view consists of the following elements:

• Harmonization of Multiple Models Ontology (H2mO): H2mO provides a formal and clear support for the most widely–used methods, concepts, relationships and related terms in harmonization of multiple models. A detailed description of H2mO ontology and its application in a real context is presented in [14].

• Process–reference Models Ontology (PrMO): is an ontology of Process–Reference Models which establishes the key elements used to express process–based approaches. From PrMO, a common structure of process elements or Common Structure of Process Elements (CSPE) has been defined, along with a homogenization technique to facilitate the harmonization of different models [15].

3.2. Methodological Framework

This describes a systematic set of activities, tasks and roles to support the efforts related to the application of a suitable strategy facilitating the harmonization of multiple models and which consists of the following elements:

• Harmonization Process (HProcess): provides a process and the elements necessary to support the systematic management and implementation of harmonization projects. A detailed description of HProcess, its activities, tasks, roles, work products, templates and other elements modelled with EPF Composer, can be seen at http://alarcos.esi.uclm.es/armonias/ and [16].

• Harmonization Methods (HMethods): is a set of methods taken as the basis for configuring a systematic harmonization strategy to be executed in order to harmonize multiple models. The harmonization strategy or HStrategy is the work product resulting from the implementation of HProcess and describes the activities to follow in order to support the harmonization of multiple models from the business objectives inside organizations. Currently, HMethods provides three methods to support the HStrategy: a Homogenization Method (HoMethod) for harmonizing the structural differences between multiple models, a comparison method (CoMethod) to identify differences and similarities between multiple models [17], and an Integration Method (IMethod) for combining and unifying best practices of multi–models. Likewise, a CSPE, which is a template defined from the process elements structure defined in PrMO to put the models into same structures, homogenizes them and makes easier both their comparison and integration.

3.3. Technological Environment

Comprises HProcessTOOL, which supports the management of harmonization projects (planning, monitoring and control), as well as their execution, by automating the techniques defined by HFramework, can be seen in [16].

4. Research methods

The methodologies guiding this project research were Action–Research and Case Studies, carried out by following the integration of these approaches. This section describes the research strategy defined for this project in terms of its roles, participants and relationships. A detailed description of the case studies, the Harmonization Framework and its process, templates and findings obtained through its implementation, are presented in [18]. We considered the following participants: Researchers' Group, Researched Object, Critical Reference Group and Stakeholders (see Table 2).

The participants in this research were (see Figure 3): the critical reference group, the researchers, and the stakeholders. The critical reference group was comprised of three case studies: case study 1, carried out in a Spanish company; case study 2, within a project for a banking sector, and case study 3, in an Italian spinoff. These case studies allowed our proposal to be validated. A more detailed description of these case studies can be found in section 5.

5. Case studies

HFramework has been applied to three harmonization projects. Table 3 highlights them and summarizes a few of their features. Case studies were carried out based on the approach presented by [19]. The design type of the case studies is ‘multiple cases' – holistic –. This is because HProcess has been implemented in three different cases where multiple models have been harmonized. The main research question to solve was related to knowing if HProcess was suitable for carrying out the harmonization of multiple reference models. A harmonization strategy or HStrategy was defined in each case study. This allowed organizing both the effort and the participants around the harmonization projects implemented in each case study. The HStrategy in each case study involved the homogenization of differences between models along with their comparison, allowing the identification of the relationships between models, and how they can complement each other. As shown in Table 3, just case study 3 needed integrated practices in order to define a new model. In this paper, we show an example of how we performed the integration of practices. A detailed description of the case studies, their HStrategies and findings, is presented in [18].

Figure 4 shows an example of the final results obtained from harmonization case 1. As it can be seen in Figure 4a, out of the 22 process areas (PAs) defined in CMMI, we found that 21 are supported by ISO models, and that only one – the Decision Analysis and Resolution (DAR) – is not. Furthermore, ISO 9001 is Largely related (76%), ISO 27001 is Partially related (22%), and ISO 20000–2 is Weakly related (2%), meaning that clauses of ISO 9001 provide greater support than other ISO models. However, as presented in Figure 4b, ISO 27001 offers support in Pas, which ISO 9001 does not address or for which it provides less support; e.g. Risk Management (RM), Measurement and Analysis (MA), and Organizational training (OT). The same is true for ISO 20000–2, which offers support in a few PAs; OT, Project Planning (PP) and Project Monitoring and Control (PMC).

Aiming at reducing the time and effort used in comparisons, we use a comparison approach applying a transitive property of equality (TPE approach); i.e. they can apply: if X(x) maps Y(y) and Y(y) maps to Z(z), then X(x) also maps to Z(z). In that sense, they can use the results of previous comparisons to establish multiple mappings; e.g. the objective in harmonization case 3 was to compare CMMI with ISO 9001, ISO 27001 and ISO 20000–2. For this matter, we have carried out the comparison between CMMI to ISO 9001, and CMMI to ISO 20000–2. Then, we have taken the results of the comparison performed in harmonization case one between ISO 27001 to ISO 20000–2, and was considered as a bridge to carry out the comparison between CMMI–Dev and ISO 27001.

We concluded: if CMMI–Dev maps to ISO 27001 and ISO 27001 maps to ISO 20000–2, then CMMI–Dev maps to ISO 20000–2. Companies can apply this rule on the description of practices stated by each model. It will allow establishing if practices of a model X and a model Z really have something in common. This simple rule could help companies, practitioners and process engineers to find relationships between multi–models from existing comparisons between models and, thus, reduce the efforts involved. Studies performed by Dirk Malzahn of OrgaTech GmbH [21] have shown that performing an assessment with this approach reduced effort by 25–40%.

In addition, the TPE approach can be applied in two ways: in practice seeing it as a cell, or on its elements seeing these as its organelles. Therefore, the type of application will depend on the level of detail the comparisons have been made with. On the other hand, it is also important to emphasize that due to the nature of TPE approach, the comparisons X(x) maps to Y(y) and Y(y) maps to Z(z) are strictly necessary, but this approach, nevertheless, will be impossible to apply.

The homogenization of model structure, identification of relationships, and integration of the models guided by HFramework and its artefacts, allowed companies to obtain successful results according to the needs of each case. In the first case study, a relationship system was defined between ISO 9001, ISO 27001, ISO 20000–2 and CMMI–DEV, which makes it possible to ascertain their mutual coverage and take advantage of their relationships and, consequently, reduce the effort involved in their application. Similarly, on the basis of results obtained and experience gained through the harmonization project, the company participating in the second case study developed a software tool for supporting and managing the transition and improvement between ISO 27001 to ISO 20000–2 [18]. In the third case study, it was possible to define an integrated IT Governance Model for Banking, which is to be applied in the Guatemala banking sector [21].

6. Supporting the integration of models

HFramework also supports the integration of models. In this regard, and on the basis of the results obtained, Table 4 presents a partial example of our unified model showing how to implement the integration of two practices. The unified practice column shows the content of a unified practice, which integrates the content of clause 8.5.3 concerning preventive action from ISO 9001:2008 and the Specific Practice (SP) of Causal Analysis and Resolution (CAR). The result is a combination of best practices into a single practice. The CMMI column indicates whether there is a relationship between the content of unified practice and CMMI. The explanation column offers additional information. The CMMI relationship column indicates that ISO clause 8.5.3 has a correspondence to CAR SPs; i.e. specific practices 1.2, 2.1, 2.2 and 2.3. Square brackets indicate information added in unified practice and angle brackets indicate deleted content.

The final result is a unified practice, which shares the quality goals of two models (see Table 5). From this type of practice, it is possible to define a multi–model process that fulfils two quality approaches. On the other hand, the institutionalization of a multi–model process makes possible to reduce the costs associated with the implementation of models by not implementing each one separately. Moreover, it allows assessment costs to be reduced for the unified requirements addressed during the ISO assessment will not be taken into account again during the CMMI assessment.

7. Lessons learned

From the results obtained after putting this proposal into practice, we have learned several lessons that are reported below, and which we believe could be taken into consideration as useful guidelines when multiple models are being harmonized.

• We think that organizations can benefit from this heterogeneity and variety if they suitably select and complement the processes, which from these models best fit their contexts.

• Several factors as the structural and terminological differences, size, approach, amongst others, impact on harmonization projects. However, they are not totally incompatible and thereby possible to be reconciled through different methods and analysis; e.g. structural differences found between specific practices of CMMI and clauses of ISO models of harmonization case three.

• There was a reduction of complexity during the homogenization, comparison and integration of models involved in the harmonization projects. This came about as a result of the definition and establishment of incremental iterations, allowing activities to be agilely managed; e.g. in harmonization case two allowed to establish short targets in each iteration, carry out supervision and regular monitoring, obtain feedback quickly, measure the progress in short periods of time, and integrate the results obtained in each iteration, continuously. Without an iterative and incremental approach, this would have been impossible.

• Management focused and directed by objectives of harmonization aligned with business needs, allows companies to obtain results according to their needs. HFramework includes activities that support the definition of a harmonization proposal based on the business necessities and the prioritized harmonization requirements..

• Applying a transitive property of equality provides companies and practitioners with a simple approach, which helps them find alternative relationships between multiple models and, thereby, extend their harmonization scope.

• There is a risk related to subjectivity when making comparisons between models. This occurs because the analysis can be influenced by the knowledge and expertise acquired with other models.

• Although a method to support the integration of models has been defined, there is a lack of a more detailed criterion to facilitate the integration in other possible situations, as well as to expedite decision making.

8. Conclusions

Currently, the wide range of models and standards provides companies with multiple solutions to choose from and decide which best fits their needs, and also brings them several benefits at different levels: security information, quality management, risks, best management practices related to technology information, amongst others. In spite of all this, it is necessary to say that due to several factors needed to be resolved before being able to have an integrated set of processes at both operational and management levels –e.g. ambiguity, incompatibility, terminology, structural differences, overlapping, amongst others–, implementing and institutionalizing multiple models is not an easy task. Following this line of thought, environments where multiple models are present characterize themselves by requiring greater effort, time and cost commitment than conventional SPI projects.

HFramework helps to resolve the structural problems between multiple models. It also supports the management and configuration of a harmonization project according to an organization business needs. It supports the harmonization of any set of models and/or standards required by one. Currently, we are replicating and refining HFramework and its elements in new harmonization projects. The main aim is to perform a study that allows us to determine whether the harmonization framework leads to a reduction in effort and costs associated with the implementation of a new model, rather than keeping one that is already institutionalized. Since this paper presents only an overview of HFramework and its application, future work will focus on a detailed presentation of case studies and experience reports along with guidelines for determining harmonization goals.

CIOs (Chief Information Officers) are becoming CPOs (Chief Process Officers); therefore, a solution that allows companies to radically address their multiple business needs through the management and improvement of their processes, along with room for rethinking, rebuilding and boosting the performance of their processes around a wide range of models, is necessary. We expect that our proposal –along with others– offers organizations the appropriate readiness to face the challenges presented by the niche markets around the world.

9. Acknowledgments

César Pardo and Francisco J. Pino acknowledge the contribution of the University of Cauca, where they work as an assistant professor and full professor respectively.

10. References

1.  J. Moore, ''An Integrated Collection of Software Engineering Standards'',IEEE Software, vol. 16, no. 6, pp. 51–57, 1999.         [ Links ]

2.  E. Oud, ''The value to IT of using international standards'', Information Systems Audit and Control Association, vol. 3, pp. 1–5, 2005.         [ Links ]

3.  I. Aaen, ''Software process improvement: Blueprints versus recipes'', IEEE Software, vol. 20, no. 5, pp. 86–93, 2003.         [ Links ]

4.  U. Andelfinger, A. Heijstek, P. Kirwan and H. Sassenburg, ''Towards a Unified Process Improvement Approach (UPIA)'', in European Software Engineering Institure Workshop (ESEPG Workshop), Frankfurt, Germany, 2006, pp. 1–6.         [ Links ]

5.  A. Ferreira, R. Machado and M. Paulk, ''Quantitative analysis of best practices models in the software domain'', in 17^th Asia Pacific Software Engineering Conference (APSEC), Sydney, Australia, 2010, pp. 433–442.         [ Links ]

6.  A. Ferreira, R. Machado and M. Paulk, ''Supporting audits and assessments in multi–model environments'', in 12^th International Conference on Product–Focused Software Process Improvement (PROFES), Torre Canne, Italy, 2011, pp. 73–87.         [ Links ]

7.  Y. Wang and G. King, Software Engineering Processes: Principles and Applications, 1^st ed. Boca Ratón, USA: CRC Press, 2000.         [ Links ]

8.  M. Paulk, ''A Comparison of ISO 9001 and the capability maturity model for software'', Software Engineering Institute/Carnegie Mellon University, Pittsburgh, USA, Tech. Rep. CMU/SEI–94–TR–12, Jul. 1994.         [ Links ]

9.  B. Mutafelija and H. Stromber, ''ISO 9001:2000 – CMMI V1.1 Mappings'', Software Engineering Institute, Boston, USA, Tech. Rep. V1.1., Dec. 2003.         [ Links ]

10.  G. Soydan and M. Kokar, ''An OWL Ontology for Representing the CMMI–SW Model'', in 2^nd International Workshop on Semantic Web Enabled Software Engineering (SWESE), Athens, Greece, 2006, pp. 1–14.         [ Links ]

11.  O. Mendes and A. Abran, ''Software engineering ontology: A development methodology'', Metrics News, vol. 9, pp. 68–76, 2004.         [ Links ]

12.  Software Engineering Institute and Carnegie Mellon University, The PrIME Project, 2015. [Online]. Available: http://www.sei.cmu.edu/process/research/prime–details.cfm. Accessed on: Feb. 19, 2014.         [ Links ]

13.  Information Systems Audit and Control Association (ISACA), Aligning COBIT 4.1, ITIL V3 and ISO/IEC 27002 for Business Benefit, 2008. [Online]. Available: http://goo.gl/E2g1e9. Accessed on: Feb. 19, 2014.         [ Links ]

14.  C. Pardo, F. Pino, F. García, M. Piattini and M. Baldassarre, ''An ontology for the harmonization of multiple standards and models'', Computer Standards & Interfaces, vol. 34, no. 1, pp. 48–59, 2012.         [ Links ]

15.  C. Pardo, F. Pino, F. García, M. Piattini and M. Baldassarre, ''A Reference Ontology for Harmonizing Process–reference Models'', Revista Facultad de Ingeniería Universidad de Antioquia, no. 73, pp. 29–42, 2014.         [ Links ]

16.  C. Pardo et al., ''HProcessTOOL: a support tool in the harmonization of multiple reference models'', in 2011 International Conference on Computational Science and Its Applications (ICCSA), Santander, Spain, 2011, pp. 370–382.         [ Links ]

17.  F. Pino, M. Baldassarre, M. Piattini, G. Visaggio and D. Caivano, ''Mapping software acquisition practices from ISO 12207 and CMMI'', in 10^th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE), Milan, Italy, 2009, pp. 234–247.         [ Links ]

18.  C. Pardo, F. Pino, F. García, M. Baldassarre and M. Piattini, ''From chaos to the systematic harmonization of multiple reference models: A harmonization framework applied in two case studies'', Journal of Systems and Software, vol. 86, no. 1, pp. 125–143, 2013.         [ Links ]

19.  R. Yin, Case Study Research: Design and Methods, 5th ed. Los Angeles, USA: Sage Publications, 2003.         [ Links ]

20.  C. Pardo et al., ''Homogenization, Comparison and Integration: A Harmonizing Strategy for the Unification of Multiple–Models in the Banking Sector'', in 12^th International Conference on Product–Focused Software Process Improvement (PROFES), Torre Canne, Italy, 2011, pp. 59–72.         [ Links ]

21.  Malzahn, ''Assessing–Learning–Improving, an Integrated Approach for Self Assessment and Process Improvement Systems'', in 4^th International Conference on Systems (ICONS), Guadeloupe, France, 2009, pp. 126–130.         [ Links ]