SciELO - Scientific Electronic Library Online

 
vol.28 número1DNA damage assessment and biological monitoring of occupational exposure to organic solvents, 2006Efficacy of the amodiaquine+sulfadoxine-pyrimethamine combination and of chloroquine for the treatment of malaria in Córdoba, Colombia, 2006 índice de autoresíndice de assuntospesquisa de artigos
Home Pagelista alfabética de periódicos  

Serviços Personalizados

Journal

Artigo

Indicadores

Links relacionados

  • Em processo de indexaçãoCitado por Google
  • Não possue artigos similaresSimilares em SciELO
  • Em processo de indexaçãoSimilares em Google

Compartilhar


Biomédica

versão impressa ISSN 0120-4157versão On-line ISSN 2590-7379

Biomédica v.28 n.1 Bogotá jan./mar. 2008

 

Cost-effectiveness of computed tomography and ultrasound in the diagnosis of appendicitis

Javier Romero1, Álvaro Sanabria2,3, Miguel Angarita1, Juan Carlos Varón4

1Department of Radiology, Fundación Santa Fe de Bogotá, Bogotá D.C., Colombia

2Department of Surgery, Universidad de la Sabana, Chía, Cundinamarca, Colombia

3 Fundación Abood Shaio, Bogotá D.C., Colombia

4 Department of Surgery, Universidad El Bosque, Bogotá D.C., Colombia

Recibido: 30/03/07; aceptado: 14/11/07

Introduction. Diagnosis of acute appendicitis is difficult in nearly 30% of patients with pain in low right quadrant. Diagnostic imaging may provide a means of a more accurate diagnosis.

Objective. The cost-effectiveness of diagnostic image tests was calculated for comparison with routine physical examination in patients with abdominal pain suspected to be appendicitis.

Materials and methods. A cost-effectiveness protocol was designed, using a decision analysis model. The standard case  was a patient with abdominal pain in right lower quadrant and suspicion of appendicitis. Three independent diagnostic alternatives were selected—ultrasonography, abdominal tomography and physical exam  in hospital with the subject under observation less than six hours. Operative characteristics, study design and costs of interventions and outcomes were assessed. The main outcome consisted of a confirmed diagnosis of appendicitis. The point of view taken was that of health maintenance organizations. Direct and indirect medical costs were measured. Time horizon used was three months. A one way sensitivity analysis was made.

Results. For a prepaid system, the most cost-effective strategy was abdominal tomography, with a cost-effectiveness ratio of US$965/diagnosed patient. For public system, the most cost-effective strategy was abdominal tomography, with a cost-effectiveness ratio of US$492/diagnosed patient.

Conclusions. Imaging diagnostic methods, in cases of abdominal pain suspected to be appendicitis, are more cost-effective than physical exam to make accurate diagnostic decisions. Tomography offers the best cost-effectiveness in prepaid system and in public health system.

Key words: appendicitis/diagnosis, cost-benefit analysis, tomography, ultrasonography, physical examination, decision support techniques.

Costo-efectividad de la tomografía computarizada y la ecografía en el diagnóstico de apendicitis

Introducción. El diagnóstico de apendicitis aguda es difícil en, al menos, 30% de los pacientes con dolor en la fosa iliaca derecha. Las imágenes diagnósticas pueden ayudar a hacer un diagnóstico más exacto.

Objetivo. Determinar el costo-efectividad de los exámenes de imágenes diagnósticas en comparación con el examen clínico en pacientes con dolor abdominal sugestivo de apendicitis.

Materiales y métodos. Se diseñó un análisis económico del tipo costo-efectividad utilizando un modelo de análisis de decisiones. El caso patrón consiste en un paciente con dolor abdominal en fosa iliaca derecha y duda diagnóstica de apendicitis. Se escogieron tres alternativas de decisión mutuamente excluyentes (ecografía, tomografía abdominal y examen clínico con observación menor de seis horas), sobre las cuales se evaluaron las características operativas, diseño de los estudios y costos de las intervenciones y sus desenlaces. El desenlace principal fue la presencia de apendicitis. El punto de vista corresponde al de la Empresas Prestadoras de Servicios de Salud. Los costos usados fueron los costos directos e indirectos médicos. Se consideró un horizonte de tiempo de tres meses. Se realizó, además, un análisis de sensibilidad a una vía.

Resultados. Para el sistema de medicina prepagada la estrategia más costo-efectiva fue la tomografía abdominal, con una razón de costo/efectividad de US$ 965/paciente diagnosticado. Para el sistema de plan obligatorio, la estrategia mas costo-efectiva fue la tomografía abdominal, con una razón costo-efectividad de US$ 491,8/paciente diagnosticado.

Conclusiones. Los métodos diagnósticos por imágenes en los casos de duda diagnóstica de apendicitis, son más costo-efectivos que el examen clínico para definir la conducta terapéutica. La tomografía axial computarizada ofrece una mejor relación costo-efectividad en el sistema de salud prepagada y es casi equiparable a la ecografía para el plan obligatorio de salud.

Palabras clave: apendicitis/diagnóstico, análisis costo-beneficio, tomografía, ultrasonografía, examen físico, técnicas de apoyo para la decisión.

Acute appendicitis is the most common surgical event in an emergency service and it is diagnosed mainly on the basis of the clinical findings. However, a substantial percentage of patients present with abdominal pain that suggests appendicitis, but for whom the clinical findings are not conclusive. Occasionally, patients with appendicitis present with atypical or associated symptoms that may mislead the examiner. Making the decision to take these patients to surgery is difficult and may result in delayed treatment or unnecessary surgery in as many as 50% of patients (1-4).

Several alternatives have been proposed as strategies to reduce the number of unnecessary surgeries, including imaging tests and clinical observation over a short period of time (5-8). Some authors have opposed the latter option with the argument that waiting creates the need to operate at a later stage, leading to increased morbidity and higher costs. Diagnostic imaging studies such as ultrasound (US) and computerized axial tomography (CT scan) provide the advantage of an almost immediate diagnosis of appendicitis, and may render the waiting time unnecessary (9-24). However, these methods may lead to significant cost increases, but with no significant benefit to the patient.

Studies conducted from an economic perspective have shown that the application of these diagnostic methods in selected patients is cost-effective (9,10,25,26). Unfortunately, conclusions from these kinds of economic studies cannot be extrapolated readily between countries because of differences in costs, and social and economic conditions (27,28). At present, no careful analysis has been conducted in Colombia regarding the cost-effectiveness of using abdominal US and CT in patients with unclear diagnosis of appendicitis. The current study was initiated to determine the cost-effectiveness of diagnostic imaging techniques under the package system and the Social Health Care Plans existing in Colombia, and to compare it with that of the standard clinical examination Additionally, the study was designed to assess operative characteristics and costs in order to determine the critical points at which imaging studies are no longer cost effective.

Materials and methods

This cost-effectiveness economic analysis was based on a decision-making model that was selected on the premise that different diagnostic strategies have different costs and different operational characteristics. Moreover, the cost-effectiveness ratio of the strategies has not been analyzed previously on the basis of a developing country cost system, and, therefore, the outcome of the strategy may lead to diagnostic recommendations that make a better use of the available resources.

For this analysis, a hypothetical population of 100 subjects was selected with the standard findings of abdominal pain in the right lower quadrant (RLQ) and clinical suspicion of appendicitis but with an uncertain diagnosis. These are cases which typically fall between the diagnostic and therapeutic thresholds. This group of patients includes mainly women of child-bearing age, who may present with atypical or unclear clinical findings. Patients with symptoms highly suggestive of appendicitis were not included in the study. For these, the treatment threshold was deemed to be sufficiently high to not require diagnostic testing before going to surgery. Also excluded were those with a low probability of having appendicitis, where the diagnostic threshold is sufficiently low to not require laboratory tests.

Decision alternatives

Pursuant to what has been accepted in the literature and to the availability of resources in Colombia, three mutually exclusive decision alternatives were chosen–appendicular ultrasound, enhanced abdominal CT scan, and clinical examination with an observation period of less than six hours–for the assessment of the following criteria:

1. Better value for the operational characteristics (sensitivity and specificity). To this end, a systematic search was conducted in the Medline database under the MESH terms “appendicitis”, “ultrasonography”, “computed tomography” and “physical exam”. This search was supplemented with a manual search of national studies published in scientific journals. For the physical exam in particular, the search was restricted to the previously defined population, namely, patients with unclear diagnosis, and excluded studies designed to assess the general population of patients with abdominal pain. This restriction was supported by the fact that paraclinical exams are required in patients with a mean probability, hence the limitation was required to maintain the same population spectrum for the clinical assessment.

2. Study design. Once the search was completed, the papers were assessed following the recommendations of the Medical Literature User Guides for diagnostic test studies. Only those that fulfilled the guidelines’ requirements (well-defined gold standard, blinded evaluation of the results, adequate patient spectrum and absence of interaction between test values and final outcome) were selected for data collection.

3. Cost of the interventions and their outcomes. Data were obtained directly from a first level institution (highly specialized institution) and from the pricing manual of the Social Security Institute.

Sequences and outcome pathways

The primary outcome was the presence or absence of appendicitis, regardless of the findings of the diagnostic tests, and effectiveness was defined as the frequency with which appendicitis was correctly diagnosed. The tree was built using the TreeAge Inc DATA 4.0 software (Figure 1).

Each pathway was assigned the probabilities of appendicitis (the criterion of diagnostic uncertainty was given the value of 0.5), the operational characteristics of each test (Table 1), and the final treatment costs. Due consideration was given to those cases where the correct diagnosis was made using any of the methods and the procedure was a simple appendectomy, whereas in those cases of diagnostic failure, the procedure was an appendectomy plus drainage of a generalized peritonitis.

Analytical perspective

The analytical perspective adopted was that of the Health Management Organizations. This perspective was chosen because these are the organizations that finally pay for the diagnostic or therapeutic approach, and therefore they need well-documented suggestions concerning the implementation of diagnostic protocols that may be included as part of the integral care modality (“package”).

In Colombia, two methods are used for contracting, each with different costs. Each was considered separately because it was not impossible to predict whether results are interchangeable between the two. The first model is called the Package system—it is more expensive but represents a relatively low coverage. The second model is the Social Health Plan—it is less costly but provides for larger coverage.

Costs

The costs with which the health management organizations are mostly concerned are the direct and indirect medical costs that determine the rate to be paid. The costs for the Package model were approximated on the basis of billing rates over a one month period and, for the Social Health Plan, obtained from the billing manual for integral care of the Social Security Institute—adjusted to 2004 prices for the Social Program model (Table 2). The surgical procedures were selected in accordance with the Procedures Manual of the Social Security Institute. Prices were converted to US dollars using the conversion rate of June 2004.

The disease was divided into two large groups in order to estimate average costs. For non-complicated appendicitis (edematous and supurative with no perforation or peritonitis), simple appendectomy was the procedure of choice. For cases of complicated appendicitis (gangrenous with perforation and localized or generalized peritonitis) laparotomy with appendectomy and overall management of the generalized peritonitis was the treatment of choice.

Time period

Appendicitis is a disease that can be considered a short-course acute event, and options for treatment are designed to avoid short-term complications. For this reason, a three-month time period was considered a sufficient time to allow for the identification of risks and benefits. Moreover, no discount adjustments were required.

Sensitivity analysis

The critical issue requiring a one-way sensitivity analysis were changes in the operative characteristics of the diagnostic test. This was particular true of ultrasound because of its operator-dependent characteristic, the sensitivity of the clinical exam as a diagnostic method, and the cost of both complicated and non-complicated appendicitis outcomes.

Results

The values of cost and probabilities used for each system are shown in table 3 and the result of the analysis of the strategies for each health care system is shown on table 4. The most cost-effective strategy for the Package system was abdominal CT scan with a cost-effectiveness ratio of $965/diagnosed patient. The incremental cost-effectiveness ratio for CT scan and ultrasound as compared with the physical exam was negative (table 3). For the Social Health Care System, CT scan was also the most cost-effective strategy, with a cost-effectiveness ratio of $491.8/diagnosed patient. The incremental cost-effectiveness ratio for CT scan and ultrasound when compared with the physical exam was also negative (table 3).

Sensitivity analysis

Within the expected ranges for the spectrum of patients with uncertain diagnosis (20-80%), the sensitivity analysis showed that, CT scan predominate as the most cost-effective method (Figure 2) for both systems.

Package health care system

The one-way sensitivity analysis showed that ultrasound is better than CT only when its sensitivity is greater than 88%. The average cost-effectiveness of CT continued to predominate in the face of cost variations of the exams and procedures.

Social Health Care Plan

The one-way sensitivity analysis showed that ultrasound is better than CT only when its sensitivity is greater than 85%. The average cost-effectiveness of CT continued to predominate in the face of cost variations of the exams and procedures.

Discussion

The diagnosis of acute appendicitis is made mainly on the basis of the clinical findings. However, a substantial proportion of patients, including women of child-bearing age, children and the elderly–nearly 30%-present with symptoms that are suggest acute appendicitis but which are bizarre or atypical in nature (3). As a consequence, the clinical exam loses diagnostic sensitivity at 50%—quite different from the 85% level accepted for the general population (1-4). This increases the number of unnecessary surgeries, the risk of complications associated with surgery, and ultimately, the cost.

Abdominal CT scan and ultrasound were proposed as means to improve diagnostic accuracy, and have a record of good performance (29-31). However, use of these technologies potentially produce increased costs for the health care system.

Studies in other countries have shown that the judicious application of these tests results in a more cost-effective strategy than the clinical exam (9,10,25,26). In spite of this evidence, the results of a cost-effectiveness study cannot be extrapolated readily, considering cost differences in different health systems (27,28).

The current study assessed the cost-effectiveness of these methods by means of a decision-making analysis model and applying the model to the two types of health care systems existant in Colombia. For both systems, the strategies based on the use of diagnostic imaging studies were better than the clinical exam alone.

In pre-paid medical care, CT scan was the predominant; The cost assessment for this system using the estimated ranges shows that the strategy using CT scan predominates in every instance. A further analysis revealed that to exceed the cost-effectiveness of a CT scan, a reduction was required of approximately 75% in the cost of care for a patient with complicated appendicitis.

Strategy with significant cost differences when compared ultrasound and the clinical exam. That predominance is lost in those cases where sensitivity of higher than 89%. This level attainable associated ultrasonographer’s experience, type health care institution test performed technology available. contrast, cost-effectiveness exam was never better CT. only greater 96%. However, not possible. Prior studies have shown highest attained, even under best conditions, 76%.>The analysis for the Social Health Plan revealed that CT scan was the best strategy. However, ultrasound may perform better than CT whenever its sensitivity is greater than 85%. This level is attainable with improved examiner training. With respect to the clinical exam, its cost-effectiveness improved when its sensitivity was greater than 95%—a situation similar to that of the pre-paid system.

The cost assessment for this system showed similar results as those reported for the pre-paid health care system.

These results generate an ethical dilemma for Colombia. Many regions are without access to diagnostic imaging technologies. In these areas, the physician must assess the case individually to determine whether the risk of a non therapeutic procedure surpasses the benefits of a diagnostic exam and then decide if the cost  of transport to a hospital with these resources is warranted. Nevertheless, clearlyt in cities with availability of CT and US, these diagnostic methods are recommended for use, particularly in patients with uncertain diagnosis. Other factors to be considered are related with the time available to perform the examination. Diagnostic imaging must be made early after admission. If this condition is not met, the potential advantages of imaging are lost.

In conclusion, the use of diagnostic imaging in uncertain cases of appendicitis is more cost-effective than the clinical exam for determining the course of action. Of the methods assessed, CT scan is the most cost-effective in both of the health care systems. The clinical exam does not appear to offer any additional benefit in assessing these patients; consequently, the current analysis recommends the implementation of these strategies in the general health system. However, institutions must confirm the values of the operative characteristics of their diagnostic tests and the cost of their interventions in order to determine their place within the ranges suggested by this study. To confirm the validity of the results presented here, experimental trials must be undertaken together with cost-effectiveness analyses. Moreover, these analyses can be more finely tuned by the consideration of opinions provided by patients and surgeons.

Conflict of interests

Financing

References

Corresponding autor:

Alvaro Sanabria, Departamento de Cirugía, Facultad de Medicina, Universidad de La Sabana, Campus Universitario

Puente del Común Km 21 via a Chía, Autopista Norte, Chía, Cundinamarca, Colombia. Tel.: (51) 8615555, ext. 2625.

alvarosanabria@gmail.com

1. Horattas MC, Guyton DP, Wu D. A reappraisal of appendicitis in the elderly. Am J Surg. 1990;160:291-3.        [ Links ]

2. Agafonoff S, Hawke I, Khadra M, Munnings V, Notaras L, Wadhwa S, et al. The influence of age and gender on normal appendicectomy rates. Aust N Z J Surg. 1987;57:843-6.        [ Links ]

3. Korner H, Sondenaa K, Soreide JA, Andersen E, Nysted A, Lende TH, et al. Incidence of acute nonperforated and perforated appendicitis: age-specific and sex-specific analysis. World J Surg. 1997;21:313-7.        [ Links ]

4. Andersson RE, Hugander A, Thulin AJ. Diagnostic accuracy and perforation rate in appendicitis: association with age and sex of the patient and with appendicectomy rate. Eur J Surg. 1992;158:37-41.        [ Links ]

5. Kirby CP, Sparnon AL. Active observation of children with possible appendicitis does not increase morbidity. ANZ J Surg. 2001;71:412-3.        [ Links ]

6. Bachoo P, Mahomed AA, Ninan GK, Youngson GG. Acute appendicitis: the continuing role for active observation. Pediatr Surg Int. 2001;17:125-8.        [ Links ]

7. Andersson RE, Hugander A, Ravn H, Offenbartl K, Ghazi SH, Nystrom PO, et al. Repeated clinical and laboratory examinations in patients with an equivocal diagnosis of appendicitis. World J Surg. 2000;24:479-85.        [ Links ]

8. Dolgin SE, Beck AR, Tartter PI. The risk of perforation when children with possible appendicitis are observed in the hospital. Surg Gynecol Obstet. 1992;175:320-4.        [ Links ]

9. Axelrod DA, Sonnad SS, Hirschl RB. An economic evaluation of sonographic examination of children with suspected appendicitis. J Pediatr Surg. 2000;35: 1236-41.        [ Links ]

10. Fujii Y, Hata J, Futagami K, Hamada T, Mitsuoka H, Teramen K, et al. Ultrasonography improves diagnostic accuracy of acute appendicitis and provides cost savings to hospitals in Japan. J Ultrasound Med. 2000;19:409-14.        [ Links ]

11. Chen SC, Chen KM, Wang SM, Chang KJ. Abdominal sonography screening of clinically diagnosed or suspected appendicitis before surgery. World J Surg. 1998;22:449-52.        [ Links ]

12. Franke C, Bohner H, Yang Q, Ohmann C, Roher HD. Ultrasonography for diagnosis of acute appendicitis: results of a prospective multicenter trial. Acute Abdominal Pain Study Group. World J Surg. 1999;23: 141-6.        [ Links ]

13. Baldisserotto M, Cavazzola S, Cavazzola LT, Lopes MH, Mottin CC. Acute edematous stump appendicitis diagnosed preoperatively on sonography. AJR Am J Roentgenol. 2000;175:503-4.        [ Links ]

14. Soda K, Nemoto K, Yoshizawa S, Hibiki T, Shizuya K, Konishi F. Detection of pinpoint tenderness on the appendix under ultrasonography is useful to confirm acute appendicitis. Arch Surg. 2001;136:1136-40.        [ Links ]

15. Stroman DL, Bayouth CV, Kuhn JA, Westmoreland M, Jones RC, Fisher TL, et al. The role of computed tomography in the diagnosis of acute appendicitis. Am J Surg. 1999;178:485-9.        [ Links ]

16. Gwynn LK. The diagnosis of acute appendicitis: clinical assessment versus computed tomography evaluation. J Emerg Med. 2001;21:119-23.        [ Links ]

17. Wise SW, Labuski MR, Kasales CJ, Blebea JS, Meilstrup JW, Holley GP, et al. Comparative assessment of CT and sonographic techniques for appendiceal imaging. AJR Am J Roentgenol. 2001;176:933-41.        [ Links ]

18. Horton MD, Counter SF, Florence MG, Hart MJ. A prospective trial of computed tomography and ultrasonography for diagnosing appendicitis in the atypical patient. Am J Surg. 2000;179:379-81.        [ Links ]

19. Lane MJ, Liu DM, Huynh MD, Jeffrey RB Jr, Mindelzun RE, Katz DS. Suspected acute appendicitis: nonenhanced helical CT in 300 consecutive patients. Radiology. 1999;213:341-6.        [ Links ]

20. Sivit CJ, Applegate KE, Berlin SC, Myers MT, Stallion A, Dudgeon DL, et al. Evaluation of suspected appendicitis in children and young adults: helical CT. Radiology. 2000;216:430-3.        [ Links ]

21. Weltman DI, Yu J, Krumenacker J Jr, Huang S, Moh P. Diagnosis of acute appendicitis: comparison of 5- and 10-mm CT sections in the same patient. Radiology. 2000;216:172-7.        [ Links ]

22. Wijetunga R, Tan BS, Rouse JC, Bigg-Wither GW, Doust BD. Diagnostic accuracy of focused appendiceal CT in clinically equivocal cases of acute appendicitis. Radiology. 2001;221:747-53.        [ Links ]

23. Fefferman NR, Roche KJ, Pinkney LP, Ambrosino MM, Genieser NB. Suspected appendicitis in children: focused CT technique for evaluation. Radiology. 2001;220:691-5.        [ Links ]

24. Jacobs JE, Birnbaum BA, Macari M, Megibow AJ, Israel G, Maki DD, et al. Acute appendicitis: comparison of helical CT diagnosis focused technique with oral contrast material versus nonfocused technique with oral and intravenous contrast material. Radiology. 2001;220:683-90.        [ Links ]

25. Pena BM, Taylor GA, Fishman SJ, Mandl KD. Costs and effectiveness of ultrasonography and limited computed tomography for diagnosing appendicitis in children. Pediatrics. 2000;106:672-6.        [ Links ]

26. Pena BM, Taylor GA, Lund DP, Mandl KD. Effect of computed tomography on patient management and costs in children with suspected appendicitis. Pediatrics. 1999;104:440-6.        [ Links ]

27. Bryan S, Brown J. Extrapolation of cost-effectiveness information to local settings. J Health Serv Res Policy. 1998;3:108-12.        [ Links ]

28. Drummond MF, Bloom BS, Carrin G, Hillman AL, Hutchings HC, Knill-Jones RP, et al. Issues in the cross-national assessment of health technology. Int J Technol Assess Health Care. 1992;8:671-82.        [ Links ]

29. Doria AS, Moineddin R, Kellenberger CJ, Epelman M, Beyene J, Schuh S, et al. US or CT for diagnosis of appendicitis in children and adults? A meta-analysis. Radiology. 2006;241:83-94.        [ Links ]

30. Weston AR, Jackson TJ, Blamey S. Diagnosis of appendicitis in adults by ultrasonography or computed tomography: a systematic review and meta-analysis. Int J Technol Assess Health Care. 2005;21:368-79.        [ Links ]

31. Terasawa T, Blackmore CC, Bent S, Kohlwes RJ. Systematic review: computed tomography and ultrasonography to detect acute appendicitis in adults and adolescents. Ann Intern Med. 2004;141:537-46.        [ Links ]

32. Rao PM, Feltmate CM, Rhea JT, Schulick AH, Novelline RA. Helical computed tomography in differentiating appendicitis and acute gynecologic conditions. Obstet Gynecol. 1999;93:417-21.        [ Links ]

33. Walker S, Haun W, Clark J, McMillin K, Zeren F, Gilliland T. The value of limited computed tomography with rectal contrast in the diagnosis of acute appendicitis. Am J Surg. 2000;180:450-4.        [ Links ]

34. Peck J, Peck A, Peck C, Peck J. The clinical role of noncontrast helical computed tomography in the diagnosis of acute appendicitis. Am J Surg 2000;180:133-6.        [ Links ]

35. Nakhgevany KB, Clarke LE. Acute appendicitis in women of childbearing age. Arch Surg. 1986;121:1053-5.        [ Links ]

36. Jess P, Bjerregaard B, Brynitz S, Holst-Christensen J, Kalaja E, Lund-Kristensen J. Acute appendicitis. Prospective trial concerning diagnostic accuracy and complications. Am J Surg. 1981;141:232-4.        [ Links ]

37. Gronroos JM, Gronroos P. A fertile-aged woman with right lower abdominal pain but unelevated leukocyte count and C-reactive protein. Acute appendicitis is very unlikely. Langenbecks Arch Surg. 1999;384:437-40.        [ Links ]

38. Rothrock SG, Green SM, Dobson M, Colucciello SA, Simmons CM. Misdiagnosis of appendicitis in nonpregnant women of childbearing age. J Emerg Med. 1995;13:1-8.        [ Links ]

Creative Commons License Todo o conteúdo deste periódico, exceto onde está identificado, está licenciado sob uma Licença Creative Commons