Print version ISSN 0120-4157
Biomédica vol.31 no.4 Bogotá Oct./Dec. 2011
1Grupo Sistemas Logísticos, Facultad de Ingeniería, Universidad de La Sabana, Chía, Colombia
2Departmento de Cirugía, Facultad de Medicina, Universidad de La Sabana, Chía, Colombia
3Departmento de Ingeniería Industrial, Facultad de Ingeniería, Universidad de los Andes, Bogotá, D.C., Colombia
Recibido: 03/09/10; aceptado:13/09/11
Introduction: Thyroid nodules are the most common endocrine condition treated by surgeons. The mainpurpose of the evaluation of a thyroid nodule is to rule out a carcinoma. Medical decisions concerning thyroid nodules are highly influenced by subjective beliefs.
Objective: To assess the subjective probabilities of malignancy that are assigned to the clinicalcharacteristics of a patient with a thyroid nodule in order to determine the degree of influence that these probabilities have on the final clinical suspicion of malignancy compared with objective data.
Material and methods: A bayesian analysis was designed to predict the risk of malignancy of a thyroid nodule based on the causal relationship between the demographic and clinical risk factors that are detected during the first consultation. A model with demographic and clinical variables using general surgeons as experts was developed.
Results: The highest probability of malignancy (94%) was assigned to the pooled case of a malewho was older than 60 years, with dysphonia, dysphagia, accelerated growth rate of the nodule and previous neck radiotherapy and who had a relative with thyroid cancer as well as multiple nodules that were larger than 1 cm and with hard consistency and palpable neck lymph nodes. For low risk cases in which the nodule characteristics are not suggestive of malignancy, the probability of malignancy assigned by clinicians was 33.59%; for high risk cases this was 75.54%.
Conclusion: Surgeons make diagnostic decisions based on subjective beliefs that do not necessarily correspond to the objective measures of the characteristics of the nodules.
Key words: thyroid nodule, Bayes theorem, thyroid neoplasm; models, statistical; probability; health knowledge, attitudes, practice.
Los cirujanos sobreestiman el riesgo de malignidad de los nódulos tiroideos, evaluación de los estimados subjetivos usando un análisis bayesiano
Introducción. Los nódulos tiroideos son la condición endocrina más frecuente para los cirujanos. El principio de la evaluación de un nódulo tiroideo es determinar si éste corresponde a un carcinoma. Las decisiones médicas sobre los nódulos tiroideos son influenciadas fuertemente por consideraciones subjetivas.
Objetivo. Determinar las probabilidades subjetivas asignadas a las características clínicas de un paciente con un nódulo tiroideo, para evaluar el grado de influencia de estas probabilidades en la sospecha clínica final de un proceso maligno en comparación con los datos objetivos.
Materiales y métodos. Se diseñó un análisis bayesiano para predecir el riesgo de un proceso maligno en un nódulo tiroideo, con base en la relación causal conocida de los factores clínicos y los demográficos durante la primera consulta. Se desarrolló un modelo con las variables clínicas y demográficas usando como expertos a los cirujanos.
Resultados. La mayor probabilidad de un proceso maligno (94 %) se asignó al caso clínico de un hombre mayor de 60 años, con disfonía y disfagia, nódulo de crecimiento rápido, antecedentes de radioterapia cervical y familiar con cáncer de tiroides, con nódulos múltiples, mayores de 1 cm, de consistencia dura y con adenomegalias cervicales palpables. Para los casos de bajo riesgo, con nódulos sin características de un proceso maligno, la probabilidad de éste asignada por los clínicos fue de 33,59 % y para los de alto riesgo de 75,54 %.
Conclusión. Los cirujanos toman decisiones diagnósticas basadas en creencias subjetivas que no necesariamente corresponden con los datos objetivos de las características de nódulos.
Palabras clave: nódulo tiroideo, teorema de Bayes, neoplasias de la tiroides, modelos estadísticos, probabilidad; conocimientos, actitudes y práctica en salud.
Thyroid nodules are the most common endocrine condition treated by surgeons. Five percent of individuals will have a thyroid nodule detected by physical examination, and this number can increase up to 35% if ultrasonography is used (1-3). The objective of evaluating a thyroid nodule is to determine if the nodule is a carcinoma or an adenoma. In carcinomas, a partial or total thyroidectomy is indicated. In adenomas, ultrasonographic followup is adequate.
Classically, some clinical variables have been associated with a greater risk of malignancy when present in a patient with thyroid nodules (4). An age greater than 45 years increases the risk of malignancy in 2 to 4% of patients with thyroid nodules, the male gender in 5-6%, previous radiotherapy in 1 to 7%, a family history of thyroid cancer in 8% and solitary and hard nodules larger than 4 cm and lateral lymph nodes in 6 to 8%. Progressive growth and obstructive symptoms have not been linked to an increased risk of malignancy in recent studies (5-8). The weight surgeons give to these findings, known as subjective probabilities, may influence their decision to perform a fine needle aspiration biopsy (FNAB) rather than follow the nodule over time. Subjective probabilities are the numeric expression of the experts´ beliefs about the probability of the occurrence of an event. Uncertainty in subjective probabilities is associated with the lack of sufficient data to allow a better understanding and quantification of variables that could be useful to create a predictive model (9).
Medical decisions concerning patients with thyroid nodules are highly influenced by subjective probabilities; diagnostic and therapeutic measures are taken based on these probabilities. Given the high incidence of thyroid nodules and the known increase in the diagnosis of small-sized nodules (10), it is important to assess the subjective probabilities that surgeons associate with malignancy in order to propose strategies to use resources cost-effectively.
A bayesian analysis was selected to assess the subjective probabilities that surgeons associate with a higher risk of malignancy in a thyroid nodule and which was based on the causal relationship of demographic and clinical risk factors that are usually evaluated during a consultation and physical exam. The advantage of the bayesian approach in the analysis of judgments is that it provides a systematic procedure to update subjective judgments or degrees of confidence using new information, groupings or by combining knowledge to make decisions (11). The departure point for bayesian analysis is the a priori probability that represents the degree of confidence that the researcher has in a hypothesis before considering new information. This a priori probability is combined with new information using Bayes´ theorem, which results in a probability a posteriori. Bayesian networks (or belief networks) is a methodology used to model and simulate the behavior of discrete-event systems under uncertainty (12).
The aim of this study was to assess the subjective probabilities assigned to the clinical characteristics of a patient who presented with a thyroid nodule to determine the degree of influence that these subjective probabilities have on the final clinical suspicion of malignancy and to compare this probability to real data based on disease incidence. This study only analyzed the diagnostic phase of the disease.
Material and methods
A bayesian analysis was selected to assess the surgeon´s beliefs that are associated with a higher risk of malignancy in a thyroid nodule. We followed five steps to develop the model:
1) the identification of variables with uncertainty;
2) the design of instruments to estimate subjective probabilities;
3) the selection and training of experts;
4) the quantification of subjective probabilities; and
5) the pooling of subjective probabilities into a predictive model.
The first three steps were performed by the authors. We selected general surgeons with a current practice in thyroid surgery and who participated in academic activities during the Colombian National Congress of Surgery of 2008.
To identify the variables to be included in the model, we followed the uncertainty matrix developed by Walker, et al. (13). A literature review was conducted to determine the risk factors for malignancy in a thyroid nodule, and the variables that have been reported as clinically useful were selected (6-8,14- 19). This review identified gender, age, family history of thyroid cancer, previous radiotherapy in the head and neck, the number, size and consistency of thyroid nodules, the growth rate of the nodule, dysphagia and dysphonia and the finding of lateral neck lymph nodes as clinically relevant variables related with the risk of malignancy.
The variables selected were validated with two head and neck surgeons who provided insight regarding the ease with which variables could be collected in a clinical setting. These variables were categorized into five groups according to the normal steps followed during the recording of clinical data on the medical record: demographic data (gender and age); medical history (previous neck radiotherapy, familiar cases of thyroid carcinoma), associated symptoms (dysphagia, dysphonia, the growth rate of the nodule), thyroid nodule characteristics at physical examination (number of nodules, size and consistency) and the finding of lateral neck lymph nodes, and introduced in the bayesian network (figure 1).
We considered high risk to include old age, male gender, previous neck radiotherapy, a family history of thyroid cancer, the associated symptoms of dysphagia and dysphonia, the growth rate of the nodule, a solitary nodule, size >2 cm, a hard consistency and the presence of lateral lymph nodes. This approach obtains the risk level associated with each variable, the corresponding group and the entire network.
This network included an estimate of 104 conditional probabilities. The authors designed 32 clinical cases and 20 control questions presented in individual cards to assess the subjective probabilities of the selected risk factors using a probability scale as an outcome (table 1). These cases were designed by one of the authors following the clinical path that is commonly used in the surgical community (identification of the patient, signs and symptoms, previous diseases and findings upon physical examination). Each case diverges in the value of only one variable (low or high risk) as previously defined.
To avoid overwhelming each expert with work, we assembled three focus groups with three experts per group and assigned 16 clinical cases and 10 control questions in a random pattern to each expert. Each expert was previously trained regarding the methodology and probable bias that could affect the procedure. Instructions included hierarchizing cards by risk level, from highest to lowest, estimating the probabilities at the end of each case and answering control questions first, followed by the questions for the clinical cases.
The pooling of probabilities from different experts was performed using a linear combination method.
For this analysis, the grade of expertise of a physician was calculated according to both the level of medical training and the years of clinical practice (table 2).
The pooled analysis was made using Hugin-Lite software (Hugin Expert AVS, Denmark).
The pooled a priori probabilities of different experts for the individual variables and clinical cases are shown in table 3 and table 4. table 3 shows the probability of malignancy that a surgeon assigned to a patient with a thyroid nodule in which the surgeon only knew the selected variables for each group as defined previously. For example, in table 3, row 1, a male patient who was older than 60 years with a thyroid nodule was assigned a probability of malignancy of 35 % without the surgeon knowing the medical history, the clinical characteristics of the nodule or the associated symptoms. The highest probability of malignancy (87%) was assigned to the case of fast growing nodule in a patient with dysphagia and dysphonia.
Table 4 provides the combined probabilities that a physician assigned to patients with thyroid nodules in which the surgeon knew the value of all of the clinical variables for each group as defined previously. For example, in table4, row 1, a male patient who was older than 60 years, with dysphonia and dysphagia, with multiple hard nodules that were larger than 1 cm with an accelerated growth rate and with palpable neck lymph nodes and who received neck radiotherapy and with a relative with thyroid cancer, had a probability of malignancy of 94%. For this step, the number of probabilities was sizeable, so we grouped risk factors as high or low risk for malignancy in a thyroid nodule using the classically accepted clinical criteria described in the material and methods section.
Because a priori probabilities do not necessarily consider the influence of one variable on the value of the other variables and because a priori probabilities are analyzed in isolation, it is necessary to build a bayesian network, which calculates a final probability (a posteriori) based on the previously assigned probability for each element of the net. For example, a physician could consider that an older male patient has a probability of malignancy of 35%, as seen in table 3, row 1, but when he considers the addition of new information about the characteristics of the nodule, such as a hard consistency and a size larger than 1 cm, he could assign a new probability of 80%, influenced by the new information, and without considering the previous probability, which should be used as a starting point to change the initial probability.
The bayesian network enables for updating the probabilities of malignancy when new patient data appear and combines all probabilities, including the previous one (a priori probability) and the influences that the new data have on calculating a posteriori probability. Figure2 shows the final results of the bayesian network, divided into low or high risk of malignancy groups according to the classically accepted risk factors. For the cases of low risk (right side) in which the patient´s characteristics are not suggestive of malignancy based on known clinical factors, the global probability of malignancy obtained from the network was 33.59% for patients without palpable neck lymph nodes. For the cases of high risk (left side), where patient characteristics are highly suggestive of malignancy based on known clinical factors, the global probability of malignancy obtained from the network was 75.54% for patients with palpable neck lymph nodes.
Figure2 shows the cumulative expected probability after adding each risk factor for malignancy using expert subjective probabilities, which resembles the clinical decision process. For example, the probability of malignancy of a male patient who is older than 60 years without palpable neck lymph nodes is 34%, and it increases to 48% if there are multiple nodules that are smaller than 1 cm, are hard and have a fast growth rate.
Thyroid nodules are a common condition seen in surgical consultation. With the common and indiscriminate use of ultrasonography in patients without any thyroid symptoms, most thyroid nodules are diagnosed, and most decisions regarding the management of these patients should be made by surgeons (10).
The most important factor to assess in a patient with a thyroid nodule is the risk of malignancy, for which surgery is mandatory. In some geographic areas, goiter is an endemic condition, and approximately 3 to 12% of nodules correspond to thyroid cancer (20,21). Classically, thyroid nodules have been assessed clinically to detect characteristics that could suggest a risk of malignancy. Young or older males with a solitary large and hard nodule with rapid growth associated with dysphonia or dysphagia, with a finding of neck lymph nodes and with a previous history of neck radiation or a parent who was diagnosed with thyroid cancer, are considered to have a high risk of malignancy.
A patient with these typical characteristics is not commonly found during consultation, and most patients belong to a spectrum in which some of these factors are present while others not and the surgeon must individually assess the probability of malignancy to devise an appropriate diagnostic strategy (diagnostic threshold) or to determine an appropriate therapeutic intervention (therapeutic threshold). Usually, this decision is made based on a subjective evaluation of the risk of malignancy associated with the nodule, as a grading scale based on objective probabilities using the data available in the literature is not regularly available in clinical settings. If we consider geographic areas with endemic goiter, the weight of the clinical factors on the perceived risk of malignancy could determine a more cost-effective use of diagnostic or therapeutic maneuvers in a patient with a thyroid nodule.
In these cases, some useful alternatives to subjective probabilities are decision trees or bayesian networks. Bayesian networks in particular are considered to be more useful because they consider the value of probabilities a priori to the results and allow for the introduction of new information when calculating the final probabilities (a posteriori probabilities). The bayesian networks increase the ease of updating the malignancy probabilities when new patient data are collected and connect all the probabilities, the previous a priori probability and the influence that new data have on producing a posteriori probabilities. Each observation of an event can incrementally increase or decrease the confidence on a hypothesis or model. The bayesian network tool is significantly more flexible than rule-based algorithms, which reject a hypothesis completely when it is inconsistent with prior observations. Applications of bayesian networks are found in fault diagnosis for complex systems with multiple state variables and multiple causal dependencies. The Bayes methodology can be easily used in real-time classification systems under uncertain conditions. For these reasons, we used a bayesian network to approach the problem of the perception of risk of malignancy on thyroid nodules. We only found 8 studies in this field, most of them using information from imaging tests or pathology reports. We considered laboratory information as a secondary step that should be applied after defining the risk of malignancy of the nodule (22-29).
Our results demonstrate that the perception of risk of thyroid nodules is overestimated. Raza, et al. (7), in a multivariable model from 600 patients, calculated risks of 15% and 99% when patients have either low or high risk factors, respectively. Tuttle, et al. (8) used a bayesian analysis in 1,121 patients to establish a risk of 3% and 80% in patients with low or high risk factors, respectively.
If we consider individual variables, as shown in table 3, lower risk factors, as may be seen in middle-aged female patients, are assigned a malignancy probability of 19%, a patient without risk factors, such as radiotherapy or family history, is assigned a probability of 25% or a patient with multiple nodules that are smaller than 1 cm and have a soft consistency is assigned a probability of 20%, which are clearly higher than those reported in the literature, of finding carcinoma in an index nodule, which is approximately 5 to 15%. This was corroborated by the clinical cases, in which a patient without any risk factors is assigned a probability of malignancy of 19%, and in the final results that were obtained by the network, where a patient without risk factors has a basal probability of malignancy of 33%.
If experts believe that the risk of malignancy of a thyroid nodule, even in a more favorable scenario, is as high as one in every three patients, he will use more diagnostic resources to confirm or rule out the diagnosis compared with a case with a lower perception of risk. Consequently, he will also communicate this perception to patients and other physicians, which may result in more diagnostic tests and a more aggressive therapy for patients.
Objective measures of risk as developed from cohort studies represent the real effect of some factors on the outcome. Translation of these measures toclinical practice continues to present a challenge. Many researchers have found that although there is sufficient information to make decisions, physicians do not follow recommendations and guidelines (30,31). Some authors believe that the problem is related to the impossibility of delivering the information to the final decision maker, and tools have been developed, including electronic reminders or rewards on health care systems, with good results (32,33).
However, we suggest this problem should be approached from a different point of view. We believe that decision makers make decisions based on subjective beliefs that do not necessarily correspond to the objective measures of the effect (34,35). This study demonstrated the important role that subjective probabilities play in a surgeon´s analysis of thyroid nodules. Therefore, we consider that one alternative approach must be to identify these subjective beliefs and to develop specific informative or educational strategies to ensure that clinical decision making considers objective data as well as subjective probabilities. Bayesian analysis, as proposed in this study, could be used to identify incongruence between data and subjective probabilities and allows the development of more focused interventions.
In conclusion, this study using a bayesian network assesses the beliefs of experts about the variables that predict malignancy in thyroid nodules and demonstrates that significant differences exist between subjective beliefs and objective measures of risk as described by the literature. This disagreement should be explored to improve the process of making decisions concerning patients with thyroid nodules.
Conflict of interests
Authors don´t have any conflicting interests that could affect the results of the study.
This work was supported by Universidad de La Sabana and Universidad de los Andes
Corresponding author: Álvaro Sanabria, Departamento de Cirugía, Facultad de Medicina, Universidad de La Sabana, Campus Puente del Común, Km21, Autopista Norte, Chía, Colombia Telephone: (571) 861 5555 firstname.lastname@example.org Presented at the World Congress of Thyroid Cancer, Toronto, Canada, August 9, 2009
1. Ezzat S, Sarti DA, Cain DR, Braunstein GD. Thyroid incidentalomas. Prevalence by palpation and ultrasonography. Arch Intern Med. 1994;154:1838-40. [ Links ]
2. Tan GH, Gharib H. Thyroid incidentalomas: management approaches to nonpalpable nodules discovered incidentally on thyroid imaging. Ann Intern Med. 1997;126:226-31. [ Links ]
3. Wiest PW, Hartshorne MF, Inskip PD, Crooks LA, VelaBS, Telepak RJ, et al. Thyroid palpation versus highresolution thyroid ultrasonography in the detection of nodules. J Ultrasound Med. 1998;17:487-96. [ Links ]
4. McCaffrey TV. Evaluation of the thyroid nodule. Cancer Control. 2000;7:223-8. [ Links ]
5. Mehta MP, Goetowski PG, Kinsella TJ. Radiation induced thyroid neoplasms 1920 to 1987: a vanishing problem? Int J Radiat Oncol Biol Phys. 1989;16:1471-5. [ Links ]
6. Rios A, Rodriguez JM, Canteras M, Galindo PJ, Balsalobre MD, Parrilla P. Risk factors for malignancy in multinodular goiters. Eur J Surg Oncol. 2004;30:58-62. [ Links ]
7. Raza SN, Shah MD, Palme CE, Hall FT, Eski S, Freeman JL. Risk factors for well-differentiated thyroid carcinoma in patients with thyroid nodular disease. Otolaryngol Head Neck Surg. 2008;139:21-6. [ Links ]
8. Tuttle RM, Lemar H, Burch HB. Clinical features associated with an increased risk of thyroid malignancy in patients with follicular neoplasia by fine-needle aspiration. Thyroid. 1998;8:377-83. [ Links ]
9. Druzdzel M, van der Gaag L. Elicitation of probabilities for belief networks: Combining qualitative and quantitative information. Utrecht, The Netherlands: Utrecht University; 1995.p.141-8. [ Links ]
10. Davies L, Welch HG. Increasing incidence of thyroid cancer in the United States, 1973-2002. JAMA. 2006; 295:2164-7. [ Links ]
11. Savage LJ. Elicitation of personal probabilities and expectations. J Am Stat Assoc. 1971;66:783. [ Links ]
12. Burnside ES. Bayesian networks: computer-assisted diagnosis support in radiology. Acad Radiol. 2005;12:422-30. [ Links ]
13. Walker K. Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene. J Expo Anal Environ Epidemiol. 2003;13:1-16. [ Links ]
14. Gurleyik E, Coskun O, Aslaner A. Clinical importance of solitary solid nodule of the thyroid in endemic goiter region. Indian J Med Sci. 2005;59:388-95. [ Links ]
15. bu-Eshy SA, Khan AR, Khan GM, al-Humaidi MA, al-Shehri MY, Malatani TS. Thyroid malignancy in multinodular goiter and solitary nodule. JR Coll Surg Edinb. 1995;40:310-2. [ Links ]
16. Mihailescu DV, Schneider AB. Size, number, and distribution of thyroid nodules and the risk of malignancy in radiation-exposed patients who underwent surgery. J Clin Endocrinol Metab. 2008;93:2188-93. [ Links ]
17. Raber W, Kaserer K, Niederle B, Vierhapper H. Risk factors for malignancy of thyroid nodules initially identified as follicular neoplasia by fine-needle aspiration: results of a prospective study of one hundred twenty patients. Thyroid. 2000;10:709-12. [ Links ]
18. Kumar H, Daykin J, Holder R, Watkinson JC, Sheppard MC, Franklyn JA. Gender, clinical findings, and serum thyrotropin measurements in the prediction of thyroid neoplasia in 1005 patients presenting with thyroid enlargement and investigated by fine-needle aspiration cytology. Thyroid. 1999;9:1105-9. [ Links ]
19. Belfiore A, La Rosa GL, La Porta GA, Giuffrida D, Milazzo G, Lupo L, et al. Cancer risk in patients with cold thyroid nodules: relevance of iodine intake, sex, age, and multinodularity. Am J Med. 1992;93:363-9. [ Links ]
20. Pelizzo MR, Bernante P, Toniato A, Fassina A. Frequency of thyroid carcinoma in a recent series of 539 consecutive thyroidectomies for multinodular goiter. Tumori. 1997;83:653-5. [ Links ]
21. Koh KB, Chang KW. Carcinoma in multinodular goiter. Br J Surg. 1992;79:266-7. [ Links ]
22. Liu YI, Kamaya A, Desser TS, Rubin DL. A bayesian network for differentiating benign from malignant thyroid nodules using sonographic and demographic features. AJR Am J Roentgenol. 2011;196:W598-W605. [ Links ]
23. Sato W, Hoshi K, Kawakami J, Sato K, Sugawara A, Saito Y, et al. Assisting the diagnosis of Graves´ hyperthyroidism with Bayesian-type and SOM-type neural networks by making use of a set of three routine tests and their correlation with free T4. Biomed Pharmacother. 2010;64:7-15. [ Links ]
24. Stojadinovic A, Peoples GE, Libutti SK, Henry LR, Eberhardt J, Howard RS, et al. Development of a clinical decision model for thyroid nodules. BMC Surg. 2009;9:12. [ Links ]
25. Liu YI, Kamaya A, Desser TS, Rubin DL. A controlled vocabulary to represent sonographic features of the thyroid and its application in a Bayesian network to predict thyroid nodule malignancy. Summit on Translat Bioinforma. 2009;2009:68-72. [ Links ]
26. Liu YI, Kamaya A, Desser TS, Rubin DL. A Bayesian classifier for differentiating benign versus malignant thyroid nodules using sonographic features. AMIA Annu Symp Proc. 2008;419-23. [ Links ]
27. Daskalakis A, Kostopoulos S, Spyridonos P, Glotsos D,Ravazoula P, Kardari M, et al. Design of a multi-classifier system for discriminating benign from malignant thyroid nodules using routinely H&E-stained cytological images. Comput Biol Med. 2008;38:196-203. [ Links ]
28. Hoshi K, Kawakami J, Sato W, Sato K, Sugawara A, Saito Y, et al. Assisting the diagnosis of thyroid diseases with Bayesian-type and SOM-type neural networks making use of routine test data. Chem Pharm Bull (Tokyo). 2006;54:1162-9. [ Links ]
29. Zhang G, Berardi VL. An investigation of neural networks in thyroid function diagnosis. Health Care Manag Sci. 1998;1:29-37. [ Links ]
30. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, et al. Why don´t physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282:1458-65. [ Links ]
31. Davis DA, Taylor-Vaisey A. Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ. 1997;157:408-16. [ Links ]
32. Rollman BL, Hanusa BH, Lowe HJ, Gilbert T, Kapoor WN, Schulberg HC. A randomized trial using computerized decision support to improve treatment of major depression in primary care. J Gen Intern Med. 2002;17:493-503. [ Links ]
33. Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, et al. A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Am Med Inform Assoc. 2005;12:431-7. [ Links ]
34. Ravitch MM. Subjectivity in decision making: common problems and limitations. World J Surg. 1989;13:281-6. [ Links ]
35. Murray E, Pollack L, White M, Lo B. Clinical decisionmaking: Patients´ preferences and experiences. Patient Educ Couns. 2007;65:189-96. [ Links ]