ARTÍCULO ORIGINAL

Managing and developing distributed research projects in software engineering by means of action-research

Gestión y desarrollo de proyectos de investigación distribuidos en ingeniería del software por medio de investigación-acción

Francisco J. Pino ^1*, Mario Piattini ², Guilherme Horta Travassos³

¹IDIS Research Group, University of Cauca. Calle 5 # 4 - 70. Popayán, Colombia.

³ESE Group, PESC/COPPE/UFRJ, C.P. 68511-21945-970. Rio de Janeiro, Brazil.

^*Autor de correspondencia: teléfono: + 57 + 2 + 82 09 800 ext. 2117, fax: + 57 + 2 + 82 09 810, correo eletrónico: fjpino@unicauca.edu.co (F. Pino)

(Recibido el 29 de enero de 2013. Aceptado el 5 de agosto de 2013)

Abstract

Participation in software research projects involving several organizations (research groups, enterprises, etc) scattered all around the world is an increasingly frequent phenomenon. The geographical distribution of the organizations entails the research projects development and management taking suitable Software Engineering research methods which must satisfy the acquired commitments as well as the new coming challenges. In that respect, this paper discusses how the Action-research qualitative method can be suitable for managing and developing software engineering distributed research projects. Furthermore, we propose a strategy to guide the use of Action-research in the context of distributed research projects. The application of this strategy in a research project, in which more than 10 enterprises and 27 research groups from 13 countries from Latin-America plus Portugal and Spain took part, is also illustrated. We observed that the use of the proposed strategy was able to provide the research managers with: (i) a suitable research project centralized administration, and (ii) appropriate coordination and apportioning of the research responsibilities for the research products construction and validation. It is important to highlight that the proposed strategy is a new way of applying the Action-research method in Software Engineering.

Keywords: Action research, research qualitative, distributed research projects

Cada vez es más frecuente que proyectos de investigación relacionados con software involucren diferentes organizaciones (grupos de investigación, empresas, etc.) repartidos por todo el mundo. Esta distribución geográfica implica que para el desarrollo y la gestión de estos proyectos de investigación se debe asumir un adecuado método de investigación de Ingeniería del Software que satisfaga los compromisos adquiridos, así como los desafíos adicionales que puedan surgir. En este sentido, el presente artículo analiza cómo el método de investigación cualitativo investigación-acción puede ser adecuado para gestionar y desarrollar proyectos de investigación distribuidos en el campo de ingeniería de software. Además, se propone una estrategia para orientar el uso del método de investigación-acción en el marco de proyectos de investigación realizados de manera distribuida. También se muestra en este artículo la aplicación de la estrategia propuesta en un proyecto de investigación que involucró más de 10 empresas y 27 grupos de investigación de 13 países de Iberoamérica. Se ha observado que el uso de esta estrategia proporcionó a los gestores de investigación: (i) una adecuada administración centralizada del proyecto de investigación, y (ii) una apropiada coordinación y reparto de las responsabilidades de investigación para la construcción y validación de los productos de investigación. Es importante resaltar que la estrategia propuesta es una nueva forma de aplicar el método de investigación-acción en Ingeniería de Software.

Palabras clave: Investigación-acción, investigación cualitativa, proyectos de investigación distribuidos

Introduction

The methods used by researchers to create new knowledge have changed, due to the frequency and ease of communication between them, increasing the dynamism of knowledge sharing, which had led to the growth of distributed research where practitioners conduct parallel research in their own locations and then share results [1]. This fact has led to a situation where many areas of research are growing more multidisciplinary and team-oriented in nature [2].

In the software Engineering area, participation in research projects of organizations scattered all over the world is an increasingly frequent phenomenon nowadays, given that there is a growing demand for this type of projects to be financed by administrations. However, distributed research projects have features that make them an altogether new form of collaboration; they use the global information infrastructure and they are team-oriented, task-sharing, and often cross- disciplinary [1]. Managing and developing these projects requires a different set of considerations than those required for other research projects.

On the other hand, Global Software Engineering is still immature, with a lack of empirical evaluation of methods, techniques and tools in an industrial context [3]. We consider that an aspect which would help to increase the maturity in this area is to look into the tailoring of research methods used in Software Engineering for guiding the use of such methods in the distributed context. In that respect, in this paper we propose a strategy to guide the use of the Action-research qualitative method in the context of distributed research projects. This has arisen from our own work and experience of using this method in an Ibero-American (Latin-

America plus Portugal and Spain) distributed research project. The COMPETISOFT project [4] was funded by CYTED (Programa Iberoamericano de Ciencia y Tecnología para el Desarrollo - Ibero-American Science and Technology Development Programme-), which was created in 1984 for multilateral scientific and technological cooperation and is supported by 21 Ibero-American countries. For its development, 1 national body for standardization and certification, more than 10 small software companies and 27 research groups from 13 countries in Ibero­America were involved. The COMPETISOFT project was developed with the objective of increasing the level of competitiveness of small software organizations, through the creation and dissemination of a common Methodological framework for the improvement and certification of their software processes, adapted to the typical characteristics of the Ibero-American software industry. Such framework is composed of (i) a process reference model, (ii) a process assessment method and (iii) an improvement framework for guiding the activities of implementation of improvements. In this paper we also discuss how Action-research can be suitable for managing and developing distributed research projects.

The rest of the paper is organized as follows. Initially an overview of the Action-research method is shown and then the strategy to guide the use of this method in distributed environments is described. Later a summary of the application of this strategy in a distributed research project is discussed. Finally, conclusions are drawn and future work is suggested.

Action-research overview

Action-research does not refer to a specific research method, but rather to a set of methods of the same type which share the following properties [5]: (i) Focus on action and change; (ii) Focus on a problem, (iii) An ''organic'' process model which involves systematic and interactive phases, and (iv) Participants' collaboration. Since it is not a specific method, there are many definitions of Action-research, the most important of which are:

• According to [6], it is the manner in which the required conditions are to be met, to learn from our own experiences and make them accessible to others.

• According to [7], it is the process of collecting research data by means of systematic mechanisms. The data collected refers to a current system related to an objective or system requirement; feeding the system with that data; undertaking actions by means of alternative variables selected from the system, based on the data and the hypotheses; and evaluating the results of the actions by collecting additional data.

• According to [8], it consists in the participation of all research members in studying the current problematic scenario, in an effort to improve or change it.

These definitions make it possible to deduce that Action-research has two aims: to benefit the research ''client'' and to increase the research knowledge [9]. Hence, Action-research is a collaborative type of research which seeks to make theory and practice meet, to establish a link between research and practice by means of a cyclical process. Action- Research focuses on yielding new knowledge which is useful in practice. It is gained by introducing changes and by researching into candidate solutions to different real scenarios which are relevant to a group in practice [10]. This is achieved thanks to the intervention of a researcher in the real circumstances surrounding the group. The results of these experiences must be beneficial to both the researcher and the participants. A fundamental premise regarding this kind of research is the complexity of social processes (and the use of information technologies in this type of organizations), which can be better studied by making changes to those processes and observing the effects of the changes [5].

An outline of the use of Action-research in Information Systems is provided in [11], including several examples published by different authors regarding the analysis, design and development of Information Systems, and particularly on software implementation and related processes. An introduction to the use of Action-research in Information Systems is given in [5], indicating ten Action-research forms and four characteristics which determine the way in which Action-research is used. These are as follows: Process Model (iterative, reflective, linear); structure (rigorous, fluid); typical involvement (collaborative, facilitative, expert); and primary goals (organizational development, system design, scientific knowledge, training). Seven basic strategies for achieving Action-research in Information Systems are listed in [12]: using the ''change paradigm'', establishing an agreement or formal research contract, providing a theoretical framework, planning data-collecting methods, maintaining collaboration and mutual learning between the researcher and the critical reference group, providing incentives for the performance of the typical cycle interactions and looking for the generalization of solutions.

In recent years, there is an increasing tendency towards the use of Action-research in Software Engineering to address different research topics [13]. However, the community of experts has detected some problems in its application, the causes of which are as follows [14]:

• Lack of Action-research methodology for Software Engineering.

• The consulting framework imposes an over-restrictive perspective, since it implies contractual liabilities and organizational interests that could be detrimental to the research.

All the issues indicated above may be understood to imply that this is not a rigorous research process. In addition, the literature survey on Action-research use in Software Engineering [13] shows that researchers of this area should be more rigorous when defining, applying and reporting Action-research studies in Software Engineering. In this respect, the main contribution of this paper is to propose a strategy to guide the use of Action- research in the context of distributed research projects, aiming to improve the control and rigor of the execution of such projects. There are several papers from the literature that deal with the suitability of Action-research for Information Systems and Software Engineering, namely [7, 10, 15, 16], but they do not propose a guideline for applying this method in the distributed environments.

Strategy to guide the use of action-research in distributed research projects

Since its origins, different ways of applying Action-research methodology have been developed [17]. To develop an Action-research strategy suitable for the features of distributed Software Engineering, it is necessary to analyze and adapt the different ways of organizing the steps and iterations described by this qualitative research method. In this respect, the study presented in [18] discusses various forms ofaction- research, including: Canonical action research, IS prototyping, Soft system methodology, Action science, Participant observation, Action learning, Multiview, Ethics, Clinical field work and Process consultation. In addition, other forms of action-research are also described in [19] (the dual imperatives of action-research) and in [20] (cooperative method development). We analyzed the characteristics of these action-research forms and we observed that none of these forms met the needs and characteristics for the management and execution of a distributed research project appropriately. For this reason, we have developed a research process for applying action-research, in the quest to address the needs, complexity and challenges related to this type of project.

The research process proposed for managing and developing distributed research projects by using action-research is shown in figure 1. Such a process is in keeping with several previous approaches on this research method:

• The general model of action-research proposed by [19] and the general outline presented in [21]. This approach describes how action-research involves a research cycle and a problem solving cycle, in which knowledge is applied and discovered interactively between activities with different goals and outcomes.

• The generic activities of a cycle of action- research indicated in [10]. According to this paper, action-research is an iterative process which involves researchers and practitioners acting together on a particular cycle of activities, including problem diagnosis, action intervention and reflective learning. We consider that it is possible to group the activity proposals according to other action-research approaches, like [19, 22, 23], within these three generic activities.

• Finally, the approach discussed in [19] has also been taken into account in strengthening our research cycle. Besides, we propose using and integrating other empirical research methods, such as Controlled Experiments, Case Studies, Surveys and Interviews to support the problem solving cycle.

Below the cycles of the research process proposed are described in a more detailed way.

Research cycles

We break down the generic activities of the Research cycle (to see figure 1) by using the tasks for action-research described by [19] as follows:

• Diagnosis: Identify the research theme, Analysis of relevant literature, and Plan and design the research project.

• Action: Action steps and Implement.

• Reflection: Monitor research, Evaluate in terms of research questions, and Amend plan and design.

The first research cycle is a special cycle, because the Research Manager must analyze the research goals and deliverables for the research project, together with the research experience, strengths and interests from the several participant organizations (work groups). Taking into account this analysis, the Research Manager makes a general plan and distributes the planned tasks to the different work groups of the project, aiming to fulfill the commitments acquired. In other words, during the first research cycle the Research Manager plans and designs the execution of the research project.

Subsequent research cycles are performed by each work group to tackle the research theme, in order to create the Research product that has been assigned, offering a solution to the problem addressed. Each work group plans and carries out activities (diagnosis, action and reflection) of its own research cycle to achieve the given tasks, that is to say to construct the Research product that is its responsibility. To reach the goal set for it, the work group should use a project structure: multi­cycle with bifurcation (see figure 2). This is most suitable when new research cycles are needed, due to the emergence of a new sub-problem and/ or new problem from the research work carried out during the current cycle [24].

• Research theme (area) (A): The research area of interest is described in this item.

• Research Framework (F): Analysis of theory, concepts and relevant literature about proposals that address the research area.

• Research method for research cycle (MR): The action-research method that guides the study of the area of interest, as well as the development of the Research product, is described in this section. In this respect, a set of forms to apply action-research methods is presented in [18].

•Reflection within the research cycles must be based on F, MR and A. The knowledge obtained from the study of the area of interest and the development of the Research product is used as feedback to refine and improve the research.

As far as the task of Analysis of relevant literature, is concerned, we should use the different approaches (Literature review, Literature survey, Systematic literature review, Systematic mapping study, etc.) to identify, analyze and interpret all available evidence related to the research area of interest.

During the task known as Plan and design the research project, it is important to offer an integrated view of the research work to be carried out, by describing the action-research elements, both the research cycle and the problem solving cycle. In this respect, the elements to consider in action-research are described in [8]:

• The researched object: This is the research area to be tackled, seeking to provide solutions to problems found in this area.

• Researchers: The person or group of people who actively carry out a research cycle. We have divided this group into different roles: research managers (who are responsible for the management of the distributed research project), area researchers (who are responsible for developing the research products) and fieldwork researchers (who are the field researchers responsible for the application of the research products in the critical reference group).

• Critical reference group: A group on which research is performed, in view of the fact that it has a problem that needs to be solved. In this group there are people who participate in the research (Software Engineering Professionals) and they can take part in the research process, though not as actively as the researcher.

• Research products: These are the work products generated during the research process that address both the research theme and the problem to be solved. This element has been proposed by us.

Figure 3 illustrates the relationship of these elements of action-research.

Problem solving cycles

By means of the problem solving cycles, work groups apply the Research products in the critical reference group to refine, improve and validate them, thus offering a suitable solution to the research problem.

• A problem in the real world (P): A real-world situation which enables researchers to tackle the research theme (A) from a practical viewpoint.

• A research method for the problem solving cycle (MPS): the method that guides the solution of the problem by using the Research products in a real situation. This method should allow the analysis of the research approach developed, in order to determine its validity.

• Reflection in the problem solving cycles is based on P and MPS. The knowledge discovered from application of the research product is used as feedback, to refine and improve the approach developed by means of the application of this new knowledge.

Figure 3 also shows that there are different ways of applying the research products in the critical reference group, which are related to the execution of the problem solving cycles. In this respect, [7] proposes four variations which basically depend on the characteristics of how this cycle is executed:

• Collaborative: The critical reference group puts in place the recommendations made by the researcher, and informs him or her of the results and effects.

• Empirical: The critical reference group carries out broad and systematic research into the situations and effects. This characteristic makes this variant difficult to implement.

• Experimental: This consists of evaluating the different options available to achieve an objective that exist. The main disadvantage is that the different options are difficult to measure, since they will be generally applied either in different organizations (with different characteristics, which may cloud the research results) or in one organization but at different times (the work environment may have changed).

We propose using an empirical research method for Global Software Engineering in order to: (i) strengthen and increase the rigor of the empirical variant during the execution of a problem solving cycle of this type, and (ii) address the difficulty of implementing this variant. In this respect, the main empirical methods used by the researchers in Global Software Engineering are case studies, surveys, controlled experiments and interviews [3]. Furthermore, there are studies that describe the necessary activities and guidelines to carry out these research methods in Software Engineering: [25] for case studies, [26] for controlled experiments, and [27-32] for surveys.

To show how the generic activities of the problem solving cycle (diagnosis, action and reflection) can be met from those activities described to these empirical research methods, we present table 1. Furthermore, this table presents the way of grouping the activities of case study, experiments and surveys within the three generic activities of a problem solving cycle.

The fieldwork researcher is in charge of preparing the protocol of the research method (including the field procedure) to use in the problem solving cycle, and they must consider the studies and activities described above. The protocol and/or field procedure can be distributed to the other field researchers or SE professionals in order to have the guideline with which to carry out the intervention in the critical reference groups (of other places) with the work products developed. At the end of the intervention in the critical reference group, each field researcher or SE professional reports on the work performed. This evidence is used by the area researchers of the research cycles to refine and improve the research products by means of the execution of new research cycles.

Applying action-research in a distributed research project

The research work began with the carrying out of the first research cycle, during which the research managers planned and designed the development and application of the Improvement framework. Furthermore, in the first meeting of the project, the planned tasks and responsibilities were distributed to the topic and fieldwork researchers.

The researchers involved in the development and application of the Improvement Framework were assigned as follows: (i) The research managers are the work group from the University of Castilla-La Mancha in Spain and the UNAM in Mexico; (ii) The topic researchers are the work groups from the University of Cauca in Colombia, the UNAM and the University of Castilla-La Mancha; (iii) the fieldwork researchers are the work groups from National University of La Matanza in Argentina, National University of La Planta in Argentina, Catholic University of the Maule in Chile; University of the Republic in Uruguay, Pontificial Catholic University of Peru, University of Castilla-La Mancha, UNAM and University of Cauca. The critical reference group is composed of the following small organizations: ENXENIO, STL and UAC Databases Laboratory from Spain; UNISOFT, INPUT, SERATIC, TOTEMS and SIDEM from Colombia; VEMN and BST from Argentina; PALL from Chile; and Ultrasist of Mexico. Figure 4 shows the location of the participants involved in the development and application of this component of COMPETISOFT.

The research theme or object (A) is the software process improvement area and, more precisely, the issue of how to guide the performing of the practices involved in this area for small software organizations. We identified, analyzed and interpreted the theory, concepts and relevant literature related to the research theme (F) by means of the execution of a systematic literature review on this issue [33], which was carried out during a initial conceptual research cycle. The research method (MR) used to lead the research was the approach described in the previous section. Initially, we analyzed the characteristics of the different action-research forms and we observed that none of these forms met the needs and characteristics appropriately for the development and application of this component of COMPETISOFT.

The research product to be developed is an Improvement framework through which to guide the activities involved in the execution of software process improvement projects in the small organizations context in detail. The framework should integrate different SPI practices, process, strategies and tools which should be tailored to small organizations' characteristics, aiming to offer them a complete guideline, which is useful and practical for addressing SPI projects.

The topic researchers responsible for the construction of the Improvement framework carried out several research cycles to develop the components of such framework. The third research cycle was carried out to define PmCOMPETISOFT [34], which describes the activities needed to manage and lead the process improvement initiatives in small organizations (this was a methodological research cycle). Then PmCOMPETISOFT was applied in companies of the critical reference group, with the support of their respective fieldwork researcher through problem solving cycles. In this respect, the problem to tackle (P) is the difficulty of successfully carrying out software process improvement projects in small organizations, and we have used the case study method (empirical variant) to guide execution of the problem solving cycles (MPS). The information and knowledge acquired from each problem solving cycle were registered in the respective case study reports, from which it was observed that more detailed improvement guidelines were necessary. With this feedback, other components of the Improvement Framework were identified and by means of the execution of new methodological research cycles, components such as METvalCOMPETISOFT, PfemCOMPETISOFT and the Strategy for process selection and prioritization [35] were developed. In addition, by means of the execution of new technical research cycles, software tools for supporting this framework such as GENESIS [36], EvalTOOL [37], HEPALE! [38] were also constructed. These components were applied in the small organizations, and the acquired knowledge from these problem solving cycles was once more used by the following research cycles to refine and improve these components of the Improvement framework, so as to create a new version of these. During the period of execution of the COMPETISFOT project, a continuous feedback between problem solving cycles and research cycles (and vice versa) took place. In each case, the latest versions available of the components of this framework were used as input for the execution of the ensuing problem solving cycle. An overview of the final version of the Improvement framework is described in [39].

Discussion

Some outcomes of the COMPETISOFT project (in which was used the research strategy) are presented of a quantitative way below:

• 19 training courses on the methodological framework, in which around of 500 stakeholders participated, were carried out in several Ibero-american countries.

• 14 small organizations from 7 countries used the COMPETISOFT's methodological framework to carry out their software process improvement initiatives.

• 6 doctoral thesis, 10 master thesis and 18 undergraduate works were performed in the context of this project.

• 2 books, 6 papers in journals of the JCR index, 4 papers in specialized journals, 22 papers in publishers LNCS, ACM and IEEE, 12 papers in Ibero-American journals, and 92 papers in proceedings of international conferences were published in the context of this project.

Based on the outcomes obtained, the main objective of Competisoft's project of create, disseminate and apply a methodological framework for software process improvement in Ibero-american small software development organizations was fulfilled. The methodological framework integrates different components such as processes, methodologies, strategies and tools which were developed of a distributed way by the different participants of this project. Furthermore taking into account the case studies carried out in the organizations of different Ibero-american countries, the increase of the capability of the processes to be improved by these organizations, the effort of applying the proposed framework and the benefits described by stakeholders of the project (reported in [39]); we consider that the methodological framework is suitable for leading SPI initiatives in small organizations. In this way, the results in terms of the framework developed, the training provided, the case studies carried out, the research works developed (to level doctoral, master and undergraduate) and the publications created (in general of the fulfillment of objectives of COMPETISOFT by the researchers of the project) are an indicator that the proposed research strategy was able to provide the research managers with: (i) a suitable research project centralized administration, and (ii) appropriate coordination and apportioning of the research responsibilities for the research products construction and validation.

On the other hand, the research on Global Software Engineering ought to focus more on evaluating practices, methods and techniques in this context, rather on the managerial problem-oriented lessons that have been learned [3]. In this respect, the proposed research strategy describes how to use the action-research qualitative method for managing and developing distributed research projects. Furthermore, according to the four characteristics of using action-research presented in [18], our research strategy has a process model: reflective, a structure: rigorous, a typical involvement: collaborative and facilitative, and primary goals: system design and scientific knowledge. In this respect, we analyzed the characteristics of the various forms of action-research discussed in [18] versus the characteristics of our research strategy, and based on this analysis, we consider that this research strategy can be a new form of applying action-research in Software Engineering.

Conclusions

This paper has concentrated on describing how the action-research qualitative method can be useful for managing and developing distributed research projects. This paper also sets forth a research strategy to guide the use of action- research in this type of projects, along with how it is used to develop and apply a component, the Improvement framework of the COMPETISOFT project.

In future work we aim to perform a systematic literature review on the use of action-research in Software Engineering. The aim is to keep on working on extending, refining and improving this strategy in order to develop a guideline for conducting and reporting on action-research in Software Engineering.

Acknowledgment

This work has been funded by the GEODAS-BC project (Ministerio de Economía y Competitividad and Fondo Europeo de Desarrollo Regional FEDER, TIN2012-37493-C03-01). Francisco J. Pino acknowledges the contribution of the University of Cauca, where he works as a Titular Professor.

References

1. C. Wagner, L. Staheli, R. Silberglitt, A. Wong, J. Kadtke. Linking Effectively: Learning Lessons from Successful Collaboration in Science and Technology (DB-345-OSTP). RAND's Science & Technology Policy Institute. 2002. Available on: http://www.rand.org/content/dam/rand/pubs/documented_briefings/2005/DB345.pdf. Accessed on: August 2013.         [ Links ]

2. M. Gibbons, C. Limoges, H. Nowotny, S. Schwartzman, P. Scott, M. Trow. The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies. 1^st ed. Ed. Sage Publications. London, England 1994. pp. 1-179        [ Links ]

3. D. Smite, C. Wohlin, T. Gorschek, R. Feldt. ''Empirical evidence in global software engineering: a systematic review''. Empirical Software Engineering. Vol. 15. 2010. pp. 91-118.         [ Links ]

4. H. Oktaba, F. Garcia, M. Piattini, F. Pino, C. Alquicira, F. Ruiz. ''Software Process Improvement: The COMPETISOFT Project''. IEEE Computer. Vol. 40. 2007. pp. 21-28.         [ Links ]

5. R. Baskerville. ''Investigating Information Systems with Action Research''. Communications of the Association for Information Systems. Vol. 2. 1999. pp. 1-32.         [ Links ]

6. R. McTaggart. ''Principles of Participatory Action Research''. Adult Education Quarterly. Vol. 41. 1991. pp. 168-187.         [ Links ]

7. W. French, C. Bell. Organization Development: Behavioral Science Interventions for Organization Improvement. 6^th ed. Ed. Prentice-Hall. London, England. 1999. pp. 1-343        [ Links ]

8. Y. Wadsworth. ''What is participatory Action Research?'' Action Research International (Paper 2). 1998. Available on: http://www.aral.com.au/ari/p-ywadsworth98.html. Accessed on: August 2013.         [ Links ]

9. N. Kock, F. Lau. ''Information Systems Action Research: Serving Two Demanding Masters''. Information Technology & People (special issue on Action Research in Information Systems). Vol. 14. 2001. pp. 6-11.         [ Links ]

10. D. Avison, F. Lan, M. Myers, A. Nielsen. ''Action Research''. Communications of the ACM. Vol. 42. 1999. pp. 94-97.         [ Links ]

11. F. Lau. A Review on the Use of Action Research in Information Systems Studies, in Information Systems Research: Information Systems and Qualitative Research. A. Lee, J. Liebenau, J. Degross (editors). Ed. Chapman & Hill. London, England. 1997. pp. 31-68.         [ Links ]

12. R. Baskerville, A. Wood. ''A Critical Perspective on Action Research as a Method for Information Systems Research''. Journal of Information Technology. Vol. 3. 1996. pp. 235-246.         [ Links ]

13. P. Medeiros, G. Travassos. Action research use in software engineering: An initial survey. In 3^rd International Symposium on Empirical Software Engineering and Measurement (ESEM 2009). Florida, USA. 2009. pp. 414-417.         [ Links ]

14. M. Diaz. Measurement Framework for the Definition of Software Measurement Programs in SMEs: MIS- PyME. PhD Thesis. Technologies and Information Systems Division. University of Castilla-La Mancha. Ciudad Real, Spain. 2009. pp. 1-259        [ Links ]

15. C. Seaman. ''Qualitative Methods in Empirical Studies of Software Engineering''. IEEE Transaction on Software Engineering. Vol. 25. 1999. pp. 557-572.         [ Links ]

16. M. Polo, M. Piattini, F. Ruiz. ''Using a qualitative research method for building a software maintenance methodology''. Software Practice and Experience. Vol. 32. 2002. pp. 1239-1260.         [ Links ]

17. I. Chein, S. Cook, J. Harding. ''The field of action research''. American Psychologist. Vol. 3. 1948. pp. 43-50.         [ Links ]

18. R. Baskerville, A. Wood. ''Diversity in information systems action research methods''. European Journal of Information Systems. Vol. 7. 1998. pp. 90-107.         [ Links ]

19. J. McKay, P. Marshall. ''The dual imperatives of action research''. Information Technology & People (special issue on Action Research in Information Systems). Vol. 14. 2001. pp. 46-59.         [ Links ]

20. Y. Dittrich, K. Rönkkö, J. Eriksson, C. Hansson, O. Lindeberg. ''Cooperative method development. Combining qualitative empirical research with method, technique and process improvement''. Empirical Software Engineering. Vol. 13. 2008. pp. 231-260.         [ Links ]

21. M. Chiasson, M. Germonprez, L. Mathiassen. ''Pluralist action research: a review of the information systems literature''. Information Systems Journal. Vol. 19. 2009. pp. 31-54.         [ Links ]

22. P. Checkland. ''From Framework through Experience to Learning: The Essential Nature of Action Research''. Information Systems Research: Contemporary Approaches and Emergent Traditions. H. Nissen, H. Klein, R. Hirscheim (editors). Ed. Elsevier. Amsterdam, Netherlands. 1991. pp. 397-403        [ Links ]

23. R. Davison, M. Martinsons, N. Kock. ''Principles of canonical action research''. Information Systems Journal. Vol. 14. 2004. pp. 65-86.         [ Links ]

24. J. McNiff. Action Research. Principles and Practice. 3^rd ed. Ed. Routledge. New York, USA. 2013. pp. 1-226        [ Links ]

25. P. Runeson, M. Höst. ''Guidelines for conducting and reporting case study research in software engineering''. Empirical Software Engineering. Vol. 14. 2009. pp. 131-164.         [ Links ]

26. C. Wohlin, P. Runeson, M. Höst, M. Ohlson, B. Regnell, A. Wesslén. Experimentation in Software Engineering: An Introduction. 1^st ed. Ed. Kluwer Academic Publishers. Norwell, Massachusetts, USA. 2000. pp. 1-204        [ Links ]

27. S. Pfleeger, B. Kitchenham. ''Principles of survey research: part 1: turning lemons into lemonade''. ACM SIGSOFT Software Engineering Notes. Vol. 26. 2001. pp. 16-18.         [ Links ]

28. B. Kitchenham, S. Pfleeger. ''Principles of survey research part 2: designing a survey''. ACM SIGSOFT Software Engineering Notes. Vol. 27. 2002. pp. 18-20.         [ Links ]

29. B. Kitchenham, S. Pfleeger. ''Principles of survey research: part 3: constructing a survey instrument''. ACM SIGSOFT Software Engineering Notes. Vol. 27. 2002. pp. 20-24.         [ Links ]

30. B. Kitchenham, S. Pfleeger. ''Principles of survey research part 4: questionnaire evaluation''. ACM SIGSOFT Software Engineering Notes. Vol. 27. 2002. pp. 20-23.         [ Links ]

31. B. Kitchenham, S. Pfleeger. ''Principles of survey research: part 5: populations and samples''. ACM SIGSOFT Software Engineering Notes. Vol. 27. 2002. pp. 17-20.         [ Links ]

32. B. Kitchenham, S. Pfleeger. ''Principles of survey research part 6: data analysis''. ACM SIGSOFT Software Engineering Notes. Vol. 28. 2GG3. pp. 24-27.         [ Links ]

33. F. Pino, F. Garcia, M. Piattini. ''Software Process Improvement in Small and Medium Software Enterprises: A Systematic Review''. Software Quality Journal. Vol. 16. 2GG8. pp. 237-261.         [ Links ]

34. F. Pino, J. Hurtado, J. Vidal, F. García, M. Piattini. A process for driving process improvement in VSEs. In International Conference on Software Process. Vancouver, Canada. 2GG9. pp. 342-353.         [ Links ]

35. F. Pino, F. Garcia, M. Piattini. Key processes to start software process improvement in small companies. In 24^th Annual ACM Symposium on Applied Computing. Honolulu, Hawaii, USA. 2009. pp. 509-516.         [ Links ]

36. M. Hernández, A. Florez, F. Pino, F. Garcia, M. Piattini, G. Ibargüengoitia, H. Oktaba. Supporting the Improvement Process for Small Software Enterprises through a software tool. in Software Enginering Symposium during Ninth Mexican International Conference on Computer Science. Mexicali, México. 2GG8. pp. 1-8.         [ Links ]

37. T. Martínez, F. Pino, E. León, F. García, M. Piattini. ''Supporting the Process Assessment through a Flexible Software Environment''. Communications in Computer and Information Science. Vol. 47. 2GG9. pp. 187-199.         [ Links ]

38. R. Cruz, M. Morales, M. Morgado, F. Pino, H. Oktaba, G. Ibargüengoitia, M. Piattini. Supporting the Software Process Improvement in Very Small Entities through E-learning: the HEPALE! Project. Mexican International Conference on Computer Science. Mexico City, Mexico. 2GG9. pp. 221-231.         [ Links ]

39. F. Pino, F. García, M. Piattini. ''An Integrated Framework to Guide Software Process Improvement in Small Organizations''. Communications in Computer and Information Science. Vol. 42. 2GG9. pp. 213-224.         [ Links ]