Agreement between the objective and subjective evaluation for predicting and finding a difficult airway

Héctor J. Meléndez*. Douglas Leal**, Diego Ramírez**

Segundo puesto. Concurso Luis Cerezo. XXVIII Congreso Colombiano de Anestesiología y Reanimación, 2009.

* Profesor Asociado Postgrado Anestesiología UIS MSc Grupo de Epidemiologia Clínica UIS.
** Anestesiólogo UIS

Recibido: marzo 2 de 2009. Enviado para modificaciones: junio 10 de 2009. Aceptado: febrero 16 de 2010.

ABSTRACT

Introduction. When assessing the airway, there are multiple factors that hinder the accurate prediction of the degree of difficulty for airway management. During the physical examination, disagreement among the physicians with regards to the definition and use of predictive factors of a difficult airway is crucial. Most assessments are subjective, not all the validated predictors are used and usually the assessment is made in an emergency setting with the patient in decubitus position, giving rise to disagreement in the Diagnosis of a difficult airway.

Objective. To assess the agreement between objective and subjective predictions and the finding of a difficult airway in a sitting and decubital position, by anesthesiologists and residents.

Materials and Methods. This was a trial for evaluating diagnostic technology. The estimated sample included 116 patients with 100 % follow-up. The patients were from the Santander University Hospital with a 1 - 3 ASA score. Agreement was established using kappa values. Every predictor was evaluated for performance (sensitivity, specificity, prognostic values) and finally, a logistic regression model was developed.

Results. The level of agreement between the objective and subjective evaluation of the difficult airway in both positions was poor. The Mallampatti classification gave the best kappa index (0,20). Intraobserver agreement was excellent or good. In general, all predictors were specific and low in sensitivity. Just the mouth opening and Mallampati classification in the predictive model exhibited a significant odds ratio (OR), which stresses the believe that an accurate oral cavity evaluation is an appropriate predictor for a difficult airway.

Key words: Intubation, Intratracheal, Risk Reduction, Diagnosis, Interobserver Variation (Source: MeSH, NLM)

INTRODUCTION

A difficult airway is defined as a clinical condition in which the conventionally trained anesthesiologist experiences difficulty with mask ventilation, difficulty with tracheal Intubation, or both. (1). Although the incidence of difficult airway is estimated at less than 10 % of the cases, it is obvious that not anticipating a difficult airway leads to disappointing clinical outcomes (2).

There are multiple factors that hinder the prediction of a difficult airway and big discrepancies when determining which factor or factors to consider. Despite the availability of objective measurements, many of these predictors are subjectively appraised in our daily practice, giving rise to huge discrepancies in the accurate Diagnosis of the difficult airway.

In the literature review we found that there is still some questioning about the feasibility of a true prediction and the variables selected for appraisal. Moreover, these variables exhibit low sensitivity and high specificity, a finding that is inconsistent with our goals when predicting a difficult airway (3-7). We were unable to find any studies in the local population assessing the prediction and the finding of a difficult airway.

Our key objective was to assess the level of agreement between the objective and the subjective prediction according to the position and the finding of a difficult airway under the definition adopted. The specific objectives included deter mining the prevalence of a difficult airway and the performance of each predictor. Finally, we developed a logistic regression model to assess any variables that could be linked to a better prediction of the difficult airway.

METHODOLOGY

We completed a diagnostic technology appraisal study. The patients enrolled were all ASA I to III scores, > 12 years old, scheduled for emergency and elective surgery under general anesthesia, at the Santander University Hospital. Patients with a known history of difficult airway were excluded, as well as those diagnosed with a difficult airway due to anatomical abnormalities, cervical immobility as a result of trauma, emergency surgery and those who for technical reasons we were unable to place in a sitting position.

The definition of a difficult airway was a Cormack grade 3 or 4 laryngoscopy finding or two or more Intubation attempts by an anesthesiologist. This definition was based on the description of difficult laryngoscopy and Intubation and is the one used in our graduate education program (3,8,9,10). For the purpose of this investigation, the laryngospy and Intubation were performed by expert anesthesiologist (defined as a graduate specialist).

The determination of the sample size followed the Bland and Altman guidelines (11,12), using as a benchmark the patterns of magnitude of concordance (k=0,6), significance (5 %) and power (80 %), for an estimated total of 96 patients that adjusted to 20 % probable losses, totaled 116. The variables used and measurement systems were determined according to scales and instruments that have been validated in multiple studies as predictors of a difficult airway (3,5,7-10).

All the patients included were subject to a data collection instrument and five evaluations were performed. The objective evaluation was done in a sitting position by the investigators who recorded the real values and the subjective evaluation was done both in the sitting and decubitus position by the anesthesiologist and the resident. They assessed each predictor categorically, depending on whether they got values above or below the cut point predictive for difficult air way. They then determined subjectively whether they felt it was a difficult airway or not.

Upon completion of the objective and the subjective evaluation, the patient was intubated and the laryngoscopic grade was recorded, as well as the number of attempts at Intubation. These data determined whether the airway is difficult or not, in accordance with the definition adopted (table 1).

The level of agreement was the resulting variable and it was assessed in terms of the finding of a difficult airway and its analysis with the objective and subjective evaluations of each variable. This agreement was quantified using kappa ratios and weighted kappa value, regardless of whether these were ordinal or nominal variables and its value was rated from poor (<0,20) to excellent (0,8 - 1), according to the accepted indicators (9,13,14) (table 2).

Despite having four quantitative variables objectively assessed by the investigators, i.e.: mouth opening, neck mobility, thyromental distance and sternomental distance, their analysis had to be done using a correlation index; however, the observer only judged the cut point subjectively (larger or smaller) and contrasted it to the real measurement. The same procedure was followed for the objective evaluation to determine whether the cut point was achieved or not, according to the measurement taken. These variables then became categorical and the evaluation could be done using the kappa index. Each trial was evaluated for performance in terms of sensitivity, specificity, and positive and negative prognostic values. Finally, a logistic regression was applied, with a view to develop a predictive model for the difficult airway, according to the evaluation and using the difficult airway finding as the reference method.

This trial was approved by the inter-institutional ethics committee; each patient submitted an informed consent and there was a 100 % follow-up.

RESULTS

The estimated sample size required (n=116) was recruited in a 10-month period from June 2007 through March 2008. The average age was 49 years, 52,6 % males; the BMI was 24,95. The expert assessors had an average professional experience of 11,5 years and the residents had been studying 1,8 years in average (Table 3).

General Difficult Airway Forecast and Agreement

On the basis of the laryngoscopic grade (>II), there were 5,17 % (n=6) difficult airway findings and on the basis of the attempts at orotracheal Intubation (>1) the number was 6,89 (n=8), for a difficult airway prevalence of 8,62 % (n=10) (Table 4).

The expert´s overall subjective prediction of a difficult airway in a sitting and decubital position was similar in absolute value to the difficult airway finding (8,62 %, n=10); however, when evaluating agreement, only 1,72 % (n=2) of the cases were predicted, equivalent to 20 % of all the difficult airway findings, with a poor kappa index (k=0,12). The assessment done by the resident with the patient sitting was similar to the expert´s assessment and improved 2,6 % (n=3) in the decubital position with kappa values of 0,25, considered as fair in decubital position and a poor kappa index (k=0,16) in a sitting position (table 5).

Intra and Inter-Observer Agreement based on each predictor

In general, the level of agreement between the difficult airway finding and the objective and subjective assessments for each prognostic factor was poor. Even the predictors for neck mobility, sterno-mental distance and protrusion of the mandible exhibited kappa indexes below 0. The expert - resident agreement in determining a difficult airway evidenced very similar results to those of the previous assessment, except for the resident´s observations in decubitus for mouth opening and thyromental distance whose kappa values were merely fair (k=0,20) (Table 6).

The expert - resident agreement gave excellent kappa indices (between 0,5 to 0,6) in both positions for the following variables: Mallampati score, mouth opening, thyromental distance, and protrusion of the mandible. The remaining predictors resulted in poor or fair kappa indices and there was even variability between the two positions. The intraobserver kappa index was excellent for each predictor, with values of 0,68 and 1; a very poor kappa index was achieved only when the resident´s evaluation was based on the sternomental distance (Table 6).

Prediction of individual variables

The performance evaluation of each trial was also based on the difficult airway finding. The only objective evaluation with a sensitivity of =>40 % was mouth opening; all the other tests exhibited high specificity (>85 %), low sensitivity (<20 %) and high negative prognostic values (>90 %).

The best performance in terms of sensitivity of the tests administered by the expert in both positions, showed considerable variability, with extreme values of 75 % for the thyromental distance, followed by 65 % for the Mallampatti score, and cero values for mouth opening and sternomental distance. The tests done by the resident in the two positions showed similar variability in terms of sensitivity, with extreme values from 0 % for mouth opening, sternomental distance and thyromental distance to 8,3 % for neck mobility. The Mallampati score in a sitting position was the test with the best sensitivity (61,9 %), as compared with the decubital position which was 0 % (Table 7).

When plotting the objective prediction for each variable with a difficult airway finding in the ROC-chart (Receiver Operating Characteristic), the resulting values are way below the ideal (0,75); these results are no different from those of the anesthesiologist´s and the resident´s subjective assessment in various positions, except for the expert´s prediction for the Mallampati score and the thyromental distance with area-under-the-curve values above 0,75 (Table 7).

Model

In order to accomplish one of the specific objectives, i.e. identifying a model for an enhanced prediction, univariate and multivariate analyses were done with additional logistics regression and modeled for each assessments.

In the objective evaluation in the final model, mouth opening was the only variable with a strong association to the difficult airway prediction with an OR of 4,04 (p=0,047).

In the model for the subjective evaluations done by the expert in the sitting and decubital positions, the only significant variable was the Mallampati score, with an OR of 1,9 (p=0,043) and 2,3 (p=0,031), respectively.

In the resident´s assessment, the final model just included mouth opening in the decubital position, with an OR of 6,3 (p=0,049). In the sitting position, none of the variables were predictive of a difficult airway, although the Mallampati score was close to being significant, with an OR of 2,16 (p=0,051). The ROC area-under-the-curve for each prognostic model showed values way below the ideal (Table 8 and Figure 1).

DISCUSSION

The ability to predict a difficult airway has been controversial in terms of its definition, scales, indices and individual predictors. Since there is a marked disagreement in the literature about the definition, assessment and prediction of the difficult airway (1,2,3,15), such disagreement impacts the variability in terms of its incidence, prevalence and performance of prognostic factors. No test is 100 % sensitive or 100 % specific, and according to several trials, the variability between these two indicators is practically the norm (7,10,15,16).

The difficult airway finding was 8,62 %; the level of agreement between the expert and the non-expert objective and subjective prediction of the difficult airway, was poor, with kappa indices of =0,2. The highest value was achieved by the resident only when doing the decubital evaluation, probably due to the degree of difficulty experienced in that position.

The inter-observer agreement exhibited kappa indices > 0,50 except for neck mobility and sterno-mental distance, probably because these are quantitative measurement variables. According to the objective performance of each test, the mouth opening showed the highest sensitivity (40 %); the rest of the tests exhibited a very low sensitivity (20 %).

Surprisingly enough, the maximum sensitivity (>60 %) obtained by the expert in both positions was achieved using the Mallampati classification. There was a similar finding when the resident did the evaluation in the sitting position.

As a whole, the subjective evaluation showed a very low sensitivity and a high specificity. This finding is no different from other studies. The sensitivity of the variables: mouth opening, sterno-mental distance and thyromental distance was nil for both appraisers, probably due to their weak agreement (1,8,10,17).

The 8,2 % of difficult airway finding is no different from the results by Shiga (7), 5,8 % incidence of difficult Intubation, Merah´s (18) 3,4 % of difficult laryngoscopy and Lavery´s (19) 1,5 % to 8,5 % of difficult Intubation.

With regards to the difficult airway finding, the prediction by the experts was low with no changes for the different positions; this may be because the tests are poor predictors of a difficult airway and the anatomic parameters basically don´t change for both positions probably because of an appraiser bias when doing the evaluation in a sitting position or because of a "certain level of confidence" in their daily practice for decubital evaluation.

These findings could not be matched with the literature since no studies were found referring to the trials performance in different positions, because the tests were done in the sitting position. (4,10-20). A new study is suggested making predictions in the different positions, by appraisers with the same degree of experience and independently.

The performance of objective tests is surprising: they showed less sensitivity and specificity in 80 % of the cases. Only one mouth opening less than 4 cm was more sensitive than the one done by the appraisers. We have no explanation for this finding; however, according to the literature, (7,15,21,22), we see that this variability has always been the norm and there will only be a few elements of judgment that enable us to make an appropriate prediction of the difficult airway. (4,7,10,23). The objective measurement of the sterno-mental distance <12 cm was the only variable that fail to predict the difficult airway, contrary to what Shige reported (7). Shiga defines it as the most useful variable to rule out a difficult airway, though his findings lack sufficient support.

Notwithstanding the poor performance of the tests to predict a difficult airway, according to the appraisers and positions, when developing the predictor models both mouth opening and Mallampati score stand out as significant and several trials give them a higher prognostic value (7,10,22). In the predictive model we developed, based on objective variables, only the mouth opening and the Mallampati score had significant ORs. This basically stresses the presumption that having an appropriate evaluation of the oral cavity is a good predictor of the difficult airway (7,18,22,23). The above findings allowed us to confirm our working hypothesis about the extensive variability between the subjective vs. the objective evaluation and the final finding of a difficult airway.

Our work, despite any weaknesses we may identify (for instance, evaluation by experts having the same level of experience), confirms that despite the parameters available to us for the evaluation of the difficult airway, there are still some hurdles to recognize one of our main problems and anesthesia-related cause of mortality; i.e. the unpredicted difficult airway. The "ideal" model for the identification of a difficult airway should be able to predict the result based on the least number of variables and with high sensitivity and specificity, so as to anticipate any requirements. This calls for new scales, indices, parameters and associations of variables.

Conflict of interests and financing

There are no conflicts of interests in this study and it was totally funded by the researchers.

REFERENCES

1. Caplan RA, Benumof JL, Berry FA, Blitt D Casey, Bode H Robert, Cheney Frederick W, Connis Richard T, et al. Practice guidelines for management of the difficult airway. Anesthesiology 1993;78:597-02.

2. George E, Haspel K. The Difficult Airway. International Anesthesiology Clinics. 2000; 38(3); 47-63.

3. Cattano D, Panicucci E, Paolicchi A, Forfori F, Giunta F and C. Hagberg. Risk Factors Assessment of the Difficult Airway: An Italian Survey of 1956 Patients. Anesth Analg 2000; 99; 1774-79.

4. Rosenstock C, Guillesber I, Gatke MR, Levin D, Kristensen MS, Rasmussen LS. Inter-observer agreement of tests used for prediction of difficult laryngoscopy/tracheal Intubation. Acta Anaesthesiol Scand 2005; 49: 1057-62

5. Lee Anna, Lawrence TY. Fan, Tony Gin, Monoj K Karmakar, Warnick D. Ngan Kee. A Systematic Review (Meta-Analysis) of the Accuracy of the Mallampati Tests to Predict the Difficult Airway. Anesth Analg 2006; 102:1867-78.

6. Naguib M, Samarkandi AH, Moniem AM, Mansur Eel-D,Alshaer AA, Al-Ayyaf HA, Fadin A, Alharby SW. Predictive Performance of Three Multivariate Difficult Tracheal Intubation Models: A Double-Blind, Case-Controlled Study. Anesth Analg 2006; 102:818-24.

7. Shiga T, Wajima Zen´ichiro, Inoue Tetsuo, Sakamoto Atsu. Predicting Difficult Intubation in Apparently Normal Patients. A Meta-analysis of Bedside Screening Test Performance. Anesthesiology 2005; 103:429-37

8. Society of Anesthesiologists. Practice Guidelines for Management of the Difficult Airway.An Updated Report by American Society of Anesthesiologists Task Force on Management of the Difficult Airway. Anesthesiology 2003; 98:1269-77

9. Pearce A. Evaluation of the airway and preparation for difficulty. Best practice & Research Clinical Anaesthesiology 2005 19, No. 4, 559-79.

10. Arne J., Descoins P, Fusciardi J, Ingrand P., Ferrier B.,Boudigues D, Aries J. Preoperative assessment for difficult Intubation in general and ENT surgery: predictive value of a clinical multivariate risk index. Br J Anaesthesia 1998, 80: 140-46

11. Altman DG. Practical Statistics for Medical Research. London: Chapman & Hall, 1991.

12. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1: 307-10.

13. Thompson WD, Walters SD. A reapraisal of the Kappa coefficient. J Clin Epidemiology 1998; 41(10): 949-58.

14. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics1977; 33: 159-74.

15. Delgado M. "Predicción clínica de intubación orotraqueal difícil. Predicción clínica de intubación orotraqueal, diseño difícil de una escala y evaluación de su reproducibilidad. Rev. Col. Anest 2000 XXVIII (1) 69-75.

16. Liess B.D. THE Difficult Airway Otolaryingol Clin N Am 2008 (41) 567- 80

17. Pearce A. Evaluation of the airway and preparation for difficulty. Best practice & Research Clinical Anaesthesiology 2005 19, (4); 559-79.

18. Lavery G. The Difficult Airway In Adult Critical Care. Crit Care Med 2008. 36;(7) 2163-73.

19. Hildittch W.G. Interobserver Reliability between a nurse and anaesthetist of test used for predicting difficult tracheal intubaction. Anaesthesia 2004 59: 881- 84

20. Merah n. Modified Mallampati Test, Thyromental Distance And Inter-Incisor Gap Are The Best Predictors Of Difficult Laryngoscopy In West Africans. Can J Anesth 2005. (52);3 291-96

21. Rosenblatt W. Decision Making in Airway Evaluation. The American Society of Anesthesiologists. ASA Refresher Courses in Anesthesiology: 2004 : 32 (1):179-86

22. Mace S.E. Challenges And Advances In Intubation: Airway Evaluation And Controversies With Intubation. Emerg Med Clin N Am. 2008. 26; 977-000.

23. Conelly N.R Management Of Unexpected Difficult Airway At A Teaching Institution Over 7 Year Period. Journal Of Clinical Anesthesia 2006 18: 198-04.