SciELO - Scientific Electronic Library Online

vol.31 issue59Linja: A Mobile Application Based on Minimax Strategy and Game TheoryDetermination of the Inside Diameter of Pressure Pipes for Drinking Water Systems Using Artificial Neural Networks author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand




Related links

  • On index processCited by Google
  • Have no similar articlesSimilars in SciELO
  • On index processSimilars in Google


Revista Facultad de Ingeniería

Print version ISSN 0121-1129On-line version ISSN 2357-5328

Rev. Fac. ing. vol.31 no.59 Tunja Jan./Mar. 2022  Epub May 04, 2022 


What is There About DevOps Assessment? A Systematic Mapping

¿Qué hay acerca de la evaluación de DevOps? Un mapeo sistemático

E a avaliação do DevOps? Um mapeamento sistemático

Carlos-Eduardo Orozco-Garcés1

César-Jesús Pardo-Calvache2

Yilber-Hernan Salazar-Mondragón3

1 M. Sc. (c) Universidad del Cauca (Popayán-Cauca, Colombia).

2 Ph. D. Universidad del Cauca (Popayán-Cauca, Colombia).

3 Universidad del Cauca (Popayán-Cauca, Colombia).


DevOps has been established as a framework used by software development companies seeking to set mechanisms to automate their development processes. Consequently, over the last decade, many companies have adopted DevOps to support their project’s development process and perform continuous improvement tasks to ensure that it is applied correctly. To achieve this, companies are looking for solutions that allow them to evaluate the degree of implementation of DevOps in their internal processes. In this sense, the objective of this study focuses on identifying, through a systematic mapping of the literature, the mechanisms used to assess DevOps in software development companies. According to the above, the current state of knowledge related to the proposal of processes, models, techniques, tools, and methodological guides is presented to conduct the DevOps assessment. As a result, it is noted that there are multiple methodological solutions that seek to assess DevOps; however, a high degree of heterogeneity was evidenced in the identified solutions, resulting in the need to establish a clear framework that serves as the basis for proposing a generic, structured, and unambiguous DevOps assessment model applicable to software companies.

Keywords: appraisal; assessment; development and operations; DevOps; evaluation; SLM


DevOps se ha consolidado como un marco de trabajo fuertemente utilizado por las empresas desarrolladoras de software que buscan establecer los mecanismos para automatizar sus procesos de desarrollo. Como consecuencia, durante la última década muchas empresas han optado por adoptar DevOps para apoyar el proceso de desarrollo de sus proyectos y realizan tareas de mejora continua para garantizar que es aplicado de manera correcta. Para lograrlo, las empresas buscan soluciones que permitan llevar a cabo la evaluación del grado de implementación de DevOps en sus procesos internos. En este sentido, el objetivo de este estudio se centra en identificar a través de un mapeo sistemático de la literatura cuáles son los mecanismos utilizados para evaluar DevOps en empresas desarrolladoras de software. De acuerdo con lo anterior, se presenta el estado de conocimiento actual relacionado a la propuesta de procesos, modelos, técnicas, herramientas y guías metodológicas para llevar a cabo la evaluación de DevOps. Como resultado, se observó que existen múltiples soluciones metodológicas que buscan evaluar DevOps a través de modelos, procesos y herramientas. No obstante, se evidenció un alto grado de heterogeneidad en las soluciones identificadas, resultando en la necesidad de establecer un marco de trabajo claro que sirva como base para proponer un modelo de evaluación de DevOps genérico, estructurado y sin ambigüedad que pueda ser utilizado por las empresas de software.

Palabras claves: desarrollo y operaciones; DevOps; evaluación; SLM


O DevOps se consolidou como um framework concentrado utilizado por empresas de desenvolvimento de software que buscam estabelecer mecanismos para automatizar seus processos de desenvolvimento. Como consequência, durante a última década muitas empresas optaram por adotar o DevOps para apoiar o processo de desenvolvimento de seus projetos e realizar tarefas de melhoria contínua para garantir que ele seja aplicado corretamente. Para isso, as empresas buscam soluções que lhes permitam realizar a avaliação do grau de implementação do DevOps em seus processos internos. Nesse sentido, o objetivo deste estudo concentra-se em identificar, por meio de um mapeamento sistemático da literatura pertinente, os mecanismos utilizados para avaliar o DevOps em empresas de desenvolvimento de software. De acordo com o exposto, apresenta-se o estado atual do conhecimento relacionado à proposta de processos, modelos, técnicas, ferramentas e guias metodológicos para realizar a avaliação DevOps. Como resultado, evidenciou-se que existem múltiplas soluções metodológicas que buscam avaliar o DevOps por meio de modelos, processos e ferramentas. No entanto, um alto grau de heterogeneidade é evidente nas soluções identificadas, resultando na necessidade de estabelecer uma estrutura clara que sirva de base para propor um modelo de avaliação de DevOps genérico, estruturado e inequívoco que possa ser utilizado pelas empresas.

Palavras claves: Desenvolvimento e Operações; DevOps; Avaliação; SLM


Software development companies are constantly seeking to formulate strategies and mechanisms to improve the processes that support the development of their projects due to the need of delivering high quality products and services in short time intervals [1]. To achieve this, companies dedicate their efforts to the definition, application, and continuous improvement of their processes and practices [1]. As a result, a set of solutions have emerged, which can be classified as traditional and agile; some of the traditional solutions are CMMI [2], RUP [3], and Waterfall [4]. On the other hand, among the agile solutions are Scrum [5], Lean Software [6], TDD [7], and XP [8]. In addition, hybrid solutions that seek to apply the best of both approaches have been proposed, among the best known are Scrum & XP [9], Scrumban [10], and Scrum & CMMI [11]. However, traditional and agile solutions propose elements related to the construction of software products (Dev), leaving aside practices related to operation/infrastructure (Ops), which are addressed by solutions such as ITIL [12], COBIT [13], and by standards such as ISO/IEC 20000 [14].

The advancement and improvement in the automation techniques of practices associated with the software development life cycle brought with it the emergence of frameworks for software development that integrate the best practices for Dev and Ops, also known as DevOps, which allows improving elements such as productivity, quality, and competitiveness of software development companies [15], [16]. The DevOps concept is not new, it was introduced in 2009 [17], and it arises with the objective of proposing a set of practices and activities necessary to close the existing gap between software development and operations, and in this way improve the speed of delivery of value, optimal functionalities, and excellent quality [18]. The foregoing, through mechanisms focused on the use of technology, human talent, and processes that allow the automation of all the stages involved during the development of software projects. In this sense, it can be said that DevOps focuses on promoting practices related to continuous integration [19], change management [20], automated tests [21], continuous deployment [22], continuous maintenance [23], among others. However, adopting DevOps in software companies is not an easy task [24]. To minimize the error risk in its implementation, companies must have the necessary elements to quantify and evaluate their degree of implementation during software development. This with the aim of generating a process of continuous improvement [25] that allows them to recognize enhancement opportunities permanently through the evaluation of their processes.

The article is organized as follows: Section 2 describes the methodology used to conduct the systematic literature mapping; Section 3 presents the results obtained from the mapping; Section 4 discusses the most important observations based on the results obtained, the limitations and implications of this field; Section 5 presents the conclusions and future work.


The systematic literature mapping (SLM) is a method used to identify relevant studies in an area of interest and the subsequent analysis of the information obtained from a set of criteria defined by the authors. The systematic mapping was carried out following the methodological guide proposed in [26-30], applying the following stages in an orderly manner: (i) Planning, (ii) Execution, and (iii) Documentation.

A. Planning Stage

The planning stage includes the following activities: (i) objectives and research questions; (ii) research strategy; (iii) inclusion/exclusion criteria; (iv) quality evaluation criteria, and (v) execution stage.

1) Objectives and research questions. The set of research questions was established following the Goal-Question-Metrics methodology (GQM). This approach suggests a measurement model composed of three levels of abstraction: (i) conceptual level (Objective); (ii) operational level (Question); and (iii) quantitative level (Metric). At a conceptual level, the research questions were designed in a way they are aligned with the objectives. They allowed to focus, characterize, and structure the information related to the area of interest. The research questions and its motivation can be consulted at

2) Research Strategy. To search for primary studies, combinations of the logical connectors “AND” and “OR” were applied. The search string was run on the following search engines: Google Scholar, IEEEXplore, Scopus, and SpringerLink. In addition, studies provided by experts in the field and used as gray literature were analyzed. The applied search string was: “(devops OR “develop and operation” OR “development and operation”) AND (capability OR maturity OR evaluation OR assessment OR measure OR measurement OR appraisal OR metric) AND ("reference model” OR tool OR process OR technique OR method).

3) Inclusion/Exclusion Criteria. Studies were assessed according to their title, abstract, and keywords. The studies selected as relevant were evaluated using the following criteria: (i) studies in English that propose mechanisms to assess DevOps; and (ii) studies published from 2009 (when the term DevOps was first defined [17]) to 2021 in high-impact journals, conferences, and congresses. On the other hand, studies that meet at least one of the following exclusion criteria were discarded: (i) studies that do not contribute to the DevOps assessment, (ii) studies that do not have a sufficient level of detail, (iii) discussion studies submitted as an abstract or presentation, (iv) studies without publication date, and (v) duplicate studies.

4) Quality Evaluation Criteria. To measure the quality of the primary studies, a questionnaire with a three-point scoring scale (1, 0, and -1) was defined. The criteria used for the evaluation of articles can be consulted at The sum of the score of each study forms the final score (obtaining a value between -6 and +6). The scores obtained do not represent an exclusion criterion for the primary articles, they are used as an indicator to identify which studies may have greater relevance in the future. The table presenting the quality evaluation of the studies can be consulted at the following link

5) Execution Stage. The selection of studies consisted of five iterations, one for each search source. For this, the following activities were conducted: (a) review of 7 studies corresponding to the gray literature; (b) selection of studies that meet the inclusion criteria; (c) selection of studies that answer the research questions, and (d) elimination of duplicate studies. As a result, 1211 related studies were identified, a total of 24 primary studies were obtained after applying each of the iterations.


The most relevant aspects are presented in relation to each one of the research questions defined for the SLM, and the corresponding references are presented to allow the reader to make a deeper analysis.

Q1: What is the temporal distribution of primary studies? It was identified that there is a growing interest as of 2014 in relation to the definition of proposals to assess DevOps. In 2019, the largest number of contributions was made with a total of 8 studies (33.3%) ([31, 32, 33, 34, 35, 36, 37, 38]), followed by 2020 with 5 studies (20.8%) ([39, 40, 41, 42, 43]), and 2018 with 4 studies (16.7%) ([44, 45, 46, 47]). On the other hand, in 2016 and 2017, 2 studies were conducted per year for a total of 4 studies (16.7%) ([48, 49, 50 ,51]). Finally, in 2014, 2015, and 2021, one study was conducted per year for a total of 3 studies (12.5%) ([52, 53, 54]).

Q2: What is the geographical distribution of primary studies? it was observed that most of the studies were conducted in Europe with a total of 15 (62.5%), out of which 5 ([35, 36, 44, 51, 52]) were proposed in the Netherlands, followed by Norway with 2 related studies ([31, 39]), and finally Germany, Austria, Spain, Finland, Italy, Lithuania, Portugal, and Sweden with 1 study each, for a total of 8 related studies ([32, 33, 34, 41, 42, 47, 49, 53]). On the other hand, (i) 3 studies were conducted in Africa (12.5%), out of which 2 were conducted in South Africa ([40, 45]) and 1 in Saudi Arabia ([37]); (ii) 3 related studies (12.5%) were identified in South America, out of which 2 were proposed in Colombia ([38, 43]) and 1 in Brazil ([50]); (iii) 2 related studies were identified in Asia (8.3%), and they were proposed by authors located in the geographical area of ​​Turkey, which belongs to the Asian continent ([46, 48]); and (iv) 1 related study was carried out in North America, specifically in the United States (4.2 %) ([54]).

Q3: What are the most cited primary studies? According to the results, it was possible to observe that the most cited study was [53] with a total of 137 citations, followed by [31] with 16 citations. On the other hand, [48] and [51] were cited 15 times each. [37, 44, 47] were cited 9 times each, [45] was cited 6 times, [39] was cited 4 times, [35, 38, 49] were cited 3 times, [52] was cited 2 times, [33, 36, 50] were cited 1 time each. Finally, [32, 34, 40, 41, 42, 43, 54] have not been cited because they were recently published and have not been sufficiently disseminated to be identified by the scientific community.

Q4: What are the research methodologies or instruments used in the literature? From the results it was observed that: (i) 10 papers (41.7%) ([34, 37, 39, 40, 42, 45, 47, 49, 55, 56]) carry out exploratory studies through SLM to establish the state of the art in the use of models, processes, techniques, tools, or reference frameworks for DevOps assessment; (ii) 7 studies (29.2 %) [32, 33, 48, 50, 57, 58, 54] propose solutions to assess DevOps through case studies in software companies; (iii) 4 studies (16.7%) ([35, 41, 46, 52]) propose metrics following the action-research model; (iv) 2 papers (8.3%) ([44, 51]) were applied through systematic reviews of the literature; and (v) 1 study (4.2%) ([31]) is carried out through empirical research models proposed by the authors.

Q5: What is the type of proposed solution? In relation to the type of solution, it was identified that: (i) in [49] an exploratory study analyzing different tools to assess DevOps in Small and Medium-sized Enterprises (SMEs) dedicated to software is carried out; (ii) in [38, 39, 43, 55] SLM was carried out to identify the elements to be considered for applying DevOps in software companies; (iii) in [34, 37, 40, 51] studies were carried out to know the state of the art in relation to the use of maturity models to assess DevOps; (iv) in [47, 52, 56, 57] metrics are proposed to evaluate specific practices such as construction, integration, and continuous deployment during the different stages of DevOps adoption; (v) in ([32, 45]) competence models are proposed; (vi) in ([34, 35, 37, 40, 41, 42, 44, 46, 50, 51]) maturity models are proposed; (vii) in [32] a collaboration model is proposed; (viii) in [50] an evaluation model adapted to DevOps based on SMM (Scrum Maturity Method) is proposed; (ix) in [33] a method to certify the use of DevOps best practices was applied; (x) in [31] a model to evaluate the development, security, and operations (DevSecOps) through the values ​​and principles proposed in DevOps is suggested; and (xi) in [54] a standard for the adoption of DevOps in software companies is proposed.

Q6: Have technological tools been proposed to assess DevOps? The analyzed studies were segmented into two categories: (i) studies that propose methodological solutions to assess DevOps (87.5%) ([31, 33, 34, 35, 39, 40, 41, 42, 44, 45, 46, 47, 50, 51, 52, 54, 55, 57, 58]); and (ii) studies that perform a comparative analysis of tools suggested to assess DevOps (12.5%) ([32, 48, 49]). To establish a broader state of knowledge regarding the use of tools, an exploratory study was carried out based on the methodology proposed in [59], in which a total of 13 tools developed by different companies seeking to assess DevOps were identified, some of the aspects were: (i) accessibility (A1), to find out if the tool is free to access, free with a trial period, or paid; (ii) method used for evaluation (A2), it is carried out through surveys, frameworks, consulting, reference models, or other; and (iii) objective or scope of the evaluation (A3), the tool evaluates the process, practices, activities, roles, tasks, principles, or other. In relation to accessibility (A1), it was observed that 7 tools (54%) ([60, 61, 62, 63, 64, 65, 66]) are free and offer their service through surveys or methodological guides, followed by 5 tools (38.4%) ([67, 68, 69, 70, 71]) which are paid, and 1 tool (7.6%) [72] that offers a trial period to the user and requests a subscription to access all the services it provides. Regarding the evaluation method used by the tools (A2), 6 of them (46.2%) ([60, 61, 62, 63, 65, 66]) carry out the evaluation through surveys; on the other hand, 5 tools (38.4%) ([68, 69, 70 ,71, 72]) carry out the evaluation through custom consulting, and 2 tools (15.4%) ([64, 67]) assess DevOps through methodological guides and specialized frameworks. Regarding the objective or scope of the evaluation (A3), 6 tools (46.2%) ([67, 68, 69, 70, 71, 72]) assess DevOps as a process taking into account the set of principles, values, tasks, activities, and roles carried out by a company; 5 tools (38.4%) ([60, 62, 63, 64, 65]) carry out an evaluation based on practices such as construction, integration, and continuous deployment; and 2 tools (15.4%) ([61, 66]) assess DevOps according to compliance with the Culture, Automation, Measurement, and Sharing principles (CAMS) proposed by DevOps.

Q7: What types of companies engage in the related studies? To conduct the analysis, the classification of large, medium, and small companies was used as a criterion according to the number of employees defined by the European Union in regulation N° 651/2014 [73] and it was complemented by the definition of micro-enterprise proposed in [74]. As a result, 14 studies (58.3%) ([31, 34, 37, 38, 39, 42, 43, 45, 46, 47, 48, 51, 52, 53]) were not applied in software development companies, 4 studies (16.7%) ([33, 36, 40, 50]) were applied only in large companies, 1 study (4.2 %) [35] was applied in a medium-sized company, and 1 study (4.2%) [41] belongs to an unknown category. On the other hand, studies evaluated in multiple companies were also conducted: 1 study [49] (4.2%) conducted the evaluation of their proposal in 1 medium-sized and 1 small company; 1 study (4.2%) [44] carried it out in a medium and a large company; 1 study (4.2%) [32] carried out multiple case studies in 3 large, 3 medium, and 3 small companies; and 1 study (4.2%) [54] proposes a standard that can be applied transversally in companies of any type.


This section presents the analysis of the results obtained from the execution of the SLM.

A. Main Observations

During the last decade, significant advances have been made in favor of defining methodological solutions and tools to assess DevOps in software companies. However, a high degree of heterogeneity was evidenced in the proposed solutions since there is no clear consensus on the definitions and concepts associated with DevOps [38]. Proposals such as [35, 41, 42, 46, 55], suggest capability and maturity models supported by the process elements proposed by CMMI, unlike [58], which follows the process elements proposed by ITIL. On the other hand, in [50] a maturity model supported by the set of values ​​and good practices proposed by SMM is suggested, and in [54] a standard for the adoption of DevOps is proposed. Also, it was identified that the industry has focused its efforts on the implementation of tools to assess DevOps through instruments such as surveys, frameworks, methodological guides, and tailored consulting services. However, each company establishes its own evaluation criteria based on the set of DevOps practices or elements that they consider appropriate. As a result, DevOps assessment solutions according to values, principles, activities, practices, roles, and tasks have been proposed. However, each author or company defines its own evaluation criteria, as they do not have a general reference model/standard to be applied transversally, and although it is clear that all solutions follow the same objective to evaluate the capacity, maturity and/or degree of competence/implementation of DevOps, there is no general consensus on how to assess DevOps in a clear and unambiguous way, thus generating confusion. Hence, a company can obtain different results after applying multiple evaluations to the same process. On the other hand, a strong interest in the use and validation of the proposed solutions in companies of different sizes was identified, focusing most of the efforts on making case studies applied to large and medium-sized companies, leaving aside the micro and small software companies, perhaps due to (i) the companies have not contemplated the institutionalization of practices related to DevOps, and/or (ii) they probably do not have the necessary resources in terms of capital and human talent that allow them to adopt assessment models optimally.

B. Limitations

The results of the exploratory study are limited to the capacity of scientific search engines. The inclusion criteria used as a starting point in the search for primary studies is limited to those written in English. In addition, the results obtained serve as a starting point for a more exhaustive version that seeks to identify gaps and elements that were not considered during this study.


In the last decade, DevOps has become one of the biggest focuses of interest for the scientific community and for the industry, which constantly seeks to improve its development processes. To achieve this, companies invest resources and time in defining good practices that allow a clear adoption of DevOps. As a result, they are in a continuous improvement process to assess whether they are applying DevOps appropriately. To do this, processes, capacity models, competence, maturity, reference frameworks, methodological guides, metrics, tools, and techniques have been proposed. However, there is a high degree of heterogeneity in these solutions, which is inconvenient for companies since they do not have a clear picture of which one, they should adopt to guarantee that their assessment is correct. In this sense, a metrics model supported by a reference model is being developed to conduct an objective DevOps assessment.


Ph. D. César Pardo and M. Sc. (c) Carlos Orozco, appreciate the contribution of Universidad del Cauca, where they work as Associate Professor and postgraduate student, respectively.


[1] H. Conradi, A. Fuggetta, “Improving software process improvement,” IEEE Software, vol. 19, no. 4, pp. 92-99, 2002. ]

[2] CMMI Institute, Capability maturity model integration for development, 2018. [ Links ]

[3] Rational Software, “Rational Unified Process,” in Best Practices in Software Development Teams, 2020 [ Links ]

[4] W. W. Royce, “Managing the development of large software systems: concepts and techniques,” in Proceedings 9th International Conference in Software Engineering, 1987, pp. 328-338 [ Links ]

[5] K. Schwaber, J. Sutherland, The scrum guide the definitive guide to scrum: The rules of the game, 2017 [ Links ]

[6] M. Poppendieck, T. Poppendieck, Lean Software Development: An Agile Toolkit. Addison-Wesley, 2003 [ Links ]

[7] K. Beck, Test Driven Development: By Example, 1st ed. Addison-Wesley Professional, 2002 [ Links ]

[8] K. Beck, E. Gamma, Extreme Programming Explained: Embrace Change, 2000 [ Links ]

[9] H. Kniberg, Scrum and XP from the Trenches, 2015 [ Links ]

[10] C. Ladas, Scrumban-essays on kanban systems for lean software development, 2009 [ Links ]

[11] J. Sutherland, C. R. Jakobsen, K. Johnson, “Scrum and CMMI level 5: The magic potion for code warriors,” in Hawaii International Conference on System Sciences, 2008, pp. 466-466 [ Links ]

[12] A. Hochstein, R. Zarnekow, W. Brenner, “ITIL as common practice reference model for IT service management: formal assessment and implications for practice,” in Conference on Electrical and Electronic Engineering, 2005, pp. 704-710 [ Links ]

[13] J. Young, G. Ridley, P. Carroll, “COBIT and Its Utilization: A Framework from the Literature,” in Hawaii International Conference on System Sciences, 2014 [ Links ]

[14] ISO/IEC, Calidad de los servicios TI,” 2019 [ Links ]

[15] M. Virmani, “Understanding Devops & Bridging The Gap From Continuous Integration To Continuous Delivery,” in The International Conference on Information and Computer Technologies, 2015, pp. 78-82 [ Links ]

[16] S. S. Samarawickrama, I. Perera, “Continuous scrum: A framework to enhance scrum with DevOps,” in International Conference on Advances in ICT for Emerging Regions, 2017, pp. 1-7 [ Links ]

[17] P. Debois, Devopsdays - Organizing Guide, 2009 [ Links ]

[18] S. Nagpal, A. Shadab, Literature Review: Promises and Challenges of DevOps, 2017 [ Links ]

[19] M. Shahin, M. A. Babar, L. Zhu, “Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices,” IEEE Access, vol. 5, pp. 3909-3943, 2017 [ Links ]

[20] C. Orozco, C. Pardo, S. Vásquez, H. Ordoñez, E. Suescún, “An agile process to support software configuration management,” RISTI, vol. 2020, no. E32, 2020 [ Links ]

[21] J. Michelsen, Dysfunction Junction: A Pragmatic Guide to Getting Started with DevOps, 2014 [ Links ]

[22] E. Diel, S. Marczak, D. S. Cruzes, “Communication Challenges and Strategies in Distributed DevOps,” in International Conference on Global Software Engineering, 2016, pp. 24-28. ]

[23] M. Soni, “End to End Automation on Cloud with Build Pipeline: The Case for DevOps in Insurance Industry, Continuous Integration, Continuous Testing, and Continuous Delivery,” in IEEE International Conference on Cloud Computing in Emerging Markets, 2015, pp. 85-89. ]

[24] J. Wettinger, V. Andrikopoulos, F. Leymann, “Automated Capturing and Systematic Usage of DevOps Knowledge for Cloud Applications,” in International Conference on Cloud Engineering, 2015, pp. 60-65 [ Links ]

[25] F. Erich, C. Amrit, M. Daneva, Report: DevOps Literature Review, 2014. ]

[26] J. D. Patón-Romero, M. Piattini, “Green IT maturity models: a systematic mapping study,” in 12 th Iberian Conference on Information Systems and Technologies, 2017, pp. 1-6 [ Links ]

[27] K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, “Systematic mapping studies in software engineering,” in International Congress on Engineering and Sustainability, 2008, pp. 1-10 [ Links ]

[28] J. Biolchini, P. G. Mian, A. C. Natali, G. H. Travassos, “Systematic review in software engineering,” in System Engineering and Computer Science Department, 2005 [ Links ]

[29] D. Budgen, M. Turner, P. Brereton, B. A. Kitchenham, “Using Mapping Studies in Software Engineering.,” in Psychology of Programming Interest Group, 2008, pp. 195-204 [ Links ]

[30] M. Genero, L. Cruz, M. Piattini, Métodos de investigación en ingeniería de software. Bogota, DC: Rama, 2014 [ Links ]

[31] N. Tomas, J. Li, H. Huang, “An empirical study on culture, automation, measurement, and sharing of devsecops,” in International Conference On Cyber Security And Protection Of Digital Services, 2019, pp. 1-8 [ Links ]

[32] P. Rittgen, S. Cronholm, H. Göbel, “Towards a Model for Assessing Collaboration Capability Between Development and Operations,” in European Systems and Software Process Improvement and Innovation, 2019, pp. 111-122 [ Links ]

[33] M. Anisetti, C. A. Ardagna, F. Gaudenzi, E. Damiani, “A Continuous Certification Methodology for DevOps,” in IEEE International Conference on Digital Ecosystems, 2019, pp. 205-212 [ Links ]

[34] M. Gasparaite, S. Ragaisis, “Comparison of devops maturity models,” in International Conference on Information Technologies, 2019, pp. 65-69 [ Links ]

[35] J. M. Radstaak, Developing a DevOps maturity model: a validated model to evaluate the maturity of DevOps in organizations, Grade Thesis, University of Twente, 2019 [ Links ]

[36] A. Caprarelli, E. Di Nitto, D. Tamburri, “Fallacies and pitfalls on the road to DevOps: a longitudinal industrial study,” in Software Engineering Aspects of Continuous Development and New Paradigms of Software Production and Deployment: Second International Workshop, 2019, pp. 200-210 [ Links ]

[37] M. Zarour, N. Alhammad, M. Alenezi, K. Alsarayrah, “A research on DevOps maturity models,” International Journal of Recent Technology and Engineering, vol. 8, no. 3, pp. 4854-4862, 2019 [ Links ]

[38] J. Guerrero, C. Certuche, K. Zúñiga, C. Pardo, “What is there about DevOps? Preliminary Findings from a Systematic Mapping Study,” in Jornadas Iberoamericanas de Ingeniería de Software e Ingeniería del Conocimiento, 2019. [ Links ]

[39] A. Mishra, Z. Otaiwi, “DevOps and software quality: A systematic mapping,” Computer Science Review, vol. 38, e100308, 2020 [ Links ]

[40] C. Marnewick and J. Langerman, “DevOps and Organisational Performance: The Fallacy of Chasing Maturity,” IEEE, 2020. [ Links ]

[41] D. Teixeira, R. Pereira, T. Henriques, M. M. Da Silva, J. Faustino, M. Silva, “A maturity model for DevOps,” International Journal of Agile Systems and Management, vol. 13, no. 4, pp. 464-511, 2020 [ Links ]

[42] T. Neubrand, T. Haendler, Development of a GQM-based Technique for Assessing DevOps Maturity, 2020 [ Links ]

[43] J. Guerrero, K. Zuñiga, C. Certuche, C. Pardo, “A systematic mapping study about DevOps,” Ciencia e Ingeniería, vol. 12, no. 1, pp. 48-62, 2020. ]

[44] R. de Feijter, S. Overbeek, R. van Vliet, E. Jagroep, S. Brinkkemper, “DevOps competences and maturity for software producing organizations,” in Enterprise, Business-Process and Information, 2018, pp. 244-259 [ Links ]

[45] T. Masombuka, E. Mnkandla, “A DevOps collaboration culture acceptance model,” in Proceedings of the American Society for Information Science and Technology, 2018, pp. 279-285. [ Links ]

[46] T. Seppä-Lassila, An assessment of DevOps maturity in a software project, Master Thesis, University of Turku, Finland, 2017 [ Links ]

[47] L. König, A. Steffens, “Towards a quality model for devops,” Continuous Software Engineering & Full-scale Software Engineering, vol. 37, 2018 [ Links ]

[48] O. E. Adalı, Ö. Özcan-Top, O. Demirörs, “Evaluation of agility assessment tools: a multiple case study,” Software Process Improvement and Capability Determination, 2016, pp. 135-149 [ Links ]

[49] M. Muñoz, J. Mejia, B. Corona, J. A. Calvo-Manzano, T. San Feliu, J. Miramontes, “Analysis of Tools for Assessing the Implementation and Use of Agile Methodologies in SMEs,” in Software Process Improvement and Capability Determination, 2016, pp. 123-134 [ Links ]

[50] R. Costa, R. Rodrigues, A. C. S. Dutra, “Application of Scrum Maturity Model in SoftDesign Company,” in Brazilian Work in Agile. Methods, 2016, pp. 39-49 [ Links ]

[51] R. Feijter, R. Vliet, E. Jagroep, S. Overbeek, S. Brinkkemper, Towards the adoption of DevOps in software product organizations: A Maturity Model Approach, 2017 [ Links ]

[52] S. Kruis, Designing a metrics model for DevOps at Philips IT, Master Thesis, Eindhoven University of Technology, 2014 [ Links ]

[53] J. Smeds, K. Nybom, I. Porres, “DevOps: A Definition and Perceived Adoption Impediments,” Lecture Notes in Business Information Processing, vol. 212, pp. 166-177, 2015. ]

[54] IEEE, “IEEE Standard for DevOps: Building Reliable and Secure Systems Including Application Build, Package, and Deployment,” IEEE Standar 2675-2021, 2021. ]

[55] G. Rong, H. Zhang, D. Shao, “CMMI guided process improvement for DevOps projects: an exploratory case study,” in Proceedings International Conference on Software Engineering, 2016, pp. 76-85 [ Links ]

[56] L. Prates, J. Faustino, M. Silva, R. Pereira, “Devsecops metrics,” in European Symposium on Systems Analysis and Design, 2019, pp. 77-90 [ Links ]

[57] P. Batra, A. Jatain, “Measurement Based Performance Evaluation of DevOps,” in International Conference on Computational Performance Ev aluation, 2020, pp. 757-760 [ Links ]

[58] M. A. McCarthy, L. M. Herger, S. M. Khan, B. M. Belgodere, “Composable DevOps: automated ontology based DevOps maturity analysis,” in International Conference on Service-Oriented Computing, 2015, pp. 600-607 [ Links ]

[59] B. Kitchenham, S. Linkman, D. Law, “DESMET: A methodology for evaluating software engineering methods and tools,” Computing and Control Engineering Journal, vol. 8, no. 3, pp. 120-126, 1997 [ Links ]

[60] ATOS, DevOps Maturity Assessment, 2021. ]

[61] Microsoft, Microsoft DevOps Self-Assessment, 2021. ]

[62] Infostretch, Infostretch DevOps Self-Assessment, 2021. ]

[63] InCycle, InCycle Evaluacion de devops, 2021. ]

[64] IBM, IBM DevOps Practice Self Assesment, 2021. ]

[65] Xmatters, DevOps Maturity Survey Report, 2021. ]

[66] Atlassian, DevOps Maturity model, 2021. ]

[67] IVI, IVI’s DevOps Assessment, 2021. ]

[68] Veritis, Veritis, 2021. ]

[69] Boxboat, Boxboat, 2021. ]

[70] Humanitec, DevOps Assessment, 2021. ]

[71] Atlassian, DevOps Assessment, 2021. ]

[72] Eficode, Eficode DevOps Assesment, 2021. ]

[73] Unión Europea, Reglamento 651/2014, 2014 [ Links ]

[74] Unión Europea, Recomendación de la comisión del 6 de mayo de 2003 sobre la definición de microempresas, pequeñas y medianas empresas, 2003 [ Links ]

Citation: C.-E. Orozco-Garcés, C.-J. Pardo-Calvache, Y.-H. Salazar-Mondragón, “What is There About DevOps Assessment? A Systematic Mapping,” Revista Facultad de Ingeniería, vol. 31 (59), e13896, 2022.


Carlos-Eduardo Orozco-Garces: Investigation, Formal Analysis, Methodology, Writing - original draft.

César-Jesús Pardo-Calvache: Supervision, Methodology, Validation, Writing-Revision and Edition.

Yilber-Hernan Salazar-Mondragón: Supervision, Methodology, Validation.

Received: December 26, 2021; Accepted: March 14, 2022; Published: March 22, 2022

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License