INTRODUCTION

Historically, the concept of evaluation and its objectives have changed in accordance with trends in education, from evaluation in the form of expert judgement, an old and still respected technique in the fields of art and medicine, to evaluation as the need to assess processes and outcomes¹. In medical education, the need to evaluate diverse contexts in areas such as clinical judgement, procedural skills, theoretical concepts and problem solving has given rise to the development of new evaluation strategies, and the Medical School Programme at Quindio University has not been an exception. To respond to this challenge, innovative evaluation strategies have been implemented in the medical programme basic area. Namely, online knowledge examination, the Interdisciplinary Evaluation and Feedback Seminar (SERI) and, more recently, the Objective Structured Clinical Examination (OSCE).

The on-line knowledge examination (Moodle®) is an attempt at solving issues with the written examination which has been the traditional strategy used in the Program, characterised by late delivery of the questions, preparation in different formats and delays in grading. With the on-line examination, a coordinator reminds faculty teachers of the date of the evaluation, the corresponding number of questions and then oversees the entry of the questions on the platform and consolidates the test. Moreover, the system provides the grading right away and the grade may be modified in case of any irregularity or correction by the faculty. This modality is in use at present, alternating with the traditional written examination.

The Interdisciplinary Evaluation and Feedback Seminar (SERI) is the result of a reflection workshop on evaluation developed by the faculty of the basic area Program in 2008². It is an individual (SERI 1) and group (SERI 2) knowledge evaluation space in which faculty of different areas get together to discuss and clarify questions pertaining to topics of previous classes, with specific questions and clinical cases. It is currently implemented in all semesters of basic training.

The OSCE was implemented as a pilot test during the second year starting on the second semester of 2010 and at the present time it is used in all semesters of basic training. It consists of five stations comprising clinical cases, simulation diagnostics, paraclinical test interpretation, and oral or written questions.

Although the evaluation strategies have been reviewed, these have been quantitative reviews as part of processes for the renewal of Colombia’s Ministry of Education registry or accreditation but not as in-depth analyses of the views of the parties directly involved, including faculty and students.

The main objective of this research was to understand innovative strategies for the evaluation of learning in the Medical Education program, from the perspective of faculty members and students.

MATERIALS AND METHODS

Design. Qualitative design with microethnographic approach that allows for a simultaneous process of theoretical construction and empirical research³. The qualitative approach was selected based on the need to conduct qualitative studies at the Health Sciences School of Quindio University, a public higher education institution with ample experience in basic and epidemiological research using the quantitative approach, and also due to the training of the researchers, obstetricians and gynaecologists and nurses, all of them faculty members with master degrees in education and experience in qualitative research. Additionally, the microethnographic approach was selected because it allows characterisation of the phenomenon under study (evaluation strategies) by means of community immersion (faculty and students of the basic área programme) based on observation and interviews with groups and individuals.

Unit of analysis. The unit of analysis used in the study consisted of innovative evaluation strategies for basic learning within the framework of the Quindio University Medical Programme. The work group was created through visits to the different groups of students of the basic area programme and clinics in order to explain the study and invite them to participate. Convenience sampling of subjects who expressed the desire to participate was used among students and faculty. Of twelve faculty members invited to participate, three had time constraints and four showed no interest.

Data collection techniques and tools. Focus groups and in-depth interviews lasting one hour were conducted by coauthor APP, using development guidelines and a field log book. Participants and interviewees gave their consent for audio recordings which were then transcribed verbatim.

Procedure. The study was developed during the second semester of 2014 and comprised five focus groups. The first focus group was attended by the five faculty members of the basic area Programme, and the remaining four were attended, respectively, by III, V, VII and IX semester students, all of them of legal age. In order to protect confidentiality, focus groups were identified as GF1 to GF5, and the participants were designated with numbers from 1 to 5. For the in-depth interviews (a total of seven), key respondents were selected according to affinity and experiences with the evaluation process (four students with high or low performance on the tests and three faculty who expressed agreement or disagreement with the strategies or who were mentioned by the students in their comments). Interviews were identified as E1 to E7. The subjects were approached in the usual teaching-learning setting. No pilot test was done considering that the process of visits to students and faculty and the selection of the participants took longer than expected. Once the initial round of focus groups and in-depth interviews was completed, and having observed that the views of the student and the faculty were consistent to a great extent, data saturation was found to exist and the data collection process was ended.

Data analysis and management. Data were managed manually by the three authors, protecting confidentiality. Verbatim transcription of the recordings was made and anonymity was ensured using a numerical code for the respondents in each focus group; likewise, only the initials of the interviewees were used during the interviews. Open axial categorisation was performed, emerging categories were identified, information sources were triangulated with a careful review of the statements from faculty members and students for each of the strategies (focus groups and interviews), and they were finally checked against the views of the authors. The report was prepared simultaneously in narrative form. In order to confirm the validity and reliability of the information obtained and in accordance with the initial commitment of returning the information to the participants, the drafted report was sent by e-mail to the students and faculty asking for comments, suggestions and their approval of the conclusions as they were understood, and no objections or suggestions for changes were received.

Bioethical considerations. All the subjects were invited to participate voluntarily after having explained the scope and objectives of the research. Verbal and written informed consents were obtained for the focus groups and for the interviews, respectively. The research study was registered as approved under Minutes No. 47 of May 13, 2013, of the Quindio University Ethics Committee.

RESULTS

Below are the results pertaining to the categories found. Several categories were derived from the analysis of the information regarding the on-line examination: Convenience - student comment: “on the on-line platform, there is agreement among almost everyone”(E2); faculty: “several subjects are evaluated on the on-line platform” (E1). Dysfunctionality - faculty: “sometimes you want to explain with words and it cannot be done” (E3); students: “it does not offer the means or the space required for students to really show their knowledge” (GF5-1). Manipulation - students: “there was fraud because it is very easy to manipulate a platform”(GF5-2); “I believe that 89% of the class cheated at some point during an on-line examination” (GF3-4). Unchangeability - students: “it is just like any other written test, so nothing changes” (GF4-1); “questions on the Moodle platform will always be the same” (GF3-2).

The following categories surfaced regarding the SERI: Approval - faculty: “they are given the opportunity to express themselves…. to make mistakes… to argue…(E3)”; students: “it allows me to see where I am failing” (E2); “I personally find it to be an interesting tool” (GF1-1). Fear - faculty: “the latest SERI was terrible, very intimidating in fact” (E1); students: “it makes you panic, to the point where your mind is blank” (GF4-4). Methodology - faculty: “you learn from your colleagues, from other faculty members” (E3); students: “the SERI helped us share knowledge and discuss our questions” (GF3-2); “the first-semester students attended one on the methodology used by doctor XXXXX and, if they have that right, we do too” (GF4-2). Grievance - students: “they think that just because they are physicians they can treat us as they please” (GF3-2); “because students are always on the losing end, and I lost” (GF2-2); faculty: “they are put under quite a nasty pressure” (E1).

Students turn to the administrative component of the Programme in the category of confrontation: “every time there is a SERI I go up to the director’s office and complain, and they always tell us we know, but nothing happens” (GF3); “I have repeatedly told Dr. XXX that just like results are expected from us, something must be expected from them as well” (GF3-4); faculty: “they almost never take the opinion into consideration” (E1).

Although the OSCE is a more recent strategy, two axial categories were identified: lack of coordination - faculty: “sometimes, our busy schedules prevent us from devoting more time to the methodology” (E3); “I have seen the same model in other universities and it worked completely different, with good coordination” (E1); students: “it needs more preparation because you can always perceive the mess” (E2); “teachers are not clear about how to organise an OSCE” (E2). Real context - faculty: “it seeks to focus on the patient … integrate everything around clinical work” (E1); student: “I had to give my diagnostic impression, and I thought it was cool” (GF2-1); “something that I like very much is that you try to think about the patient” (E2).

DISCUSSION

Portfolios⁴, progress tests⁵, and the Objective Structured Clinical Examination (OSCE)⁶ are cited among evaluation strategies different from the traditional approaches used in medical education such as oral and written tests, questions during rounds, and consultation.

The OSCE, described more than 30 years ago⁶, has reached new heights of late because of its proven effectiveness in assessing competencies. It brings together written knowledge tests, interpretation of paraclinical tests, physical examination and clinical judgement in an objective and controlled environment⁷. Students move through different stations at which they are presented with a problema situation they need to solve within a set period of time, including stations where they can rest⁸. It is believed that the greater the number of stations used, the more comprehensive the ability of the test to assess competencies⁹.

In the Medical School Programme of Quindio University, few faculty members have used portfolios, and innovation in evaluation has focused on the use of the exam on the Moodle platform, and the implementation of the SERI and of the OSCE. Having analysed and interpreted the information, this study contributes to the consolidation of the authors’ understanding regarding each of the strategies.

On-line examination. There is evidence of a process of transition towards new technologies. This fact is supported by the construction of virtual spaces by the faculty, allowing them to assess different student competencies. In this regard, García et al.¹⁰ state that, “in general terms, evaluation systems must be adapted to the learning objectives, the content and the subjects of evaluation and, if they are implemented in a virtual environment, then they must be adapted to that specific methodology.”

There is room for improvement considering the annoyance expressed by some of the participants in the study due to the manipulation of the platform and cheating by some of the students, resulting in loss of validity and objectivity of this strategy (which was actually halted temporarily for that reason). To this point, Labra¹¹ argues that “an embedded automatic system that checks for potential copy or plagiarism may be considered.” On the other hand, it is the perception of the interviewees that the contribution gained from on-line evaluation is just a more expedite grading process, they have the impression that tests are still traditional, with a predominance of memory, and that the choice of just one answer is the right one.

Interdisciplinary Evaluation and Feedback Seminar (SERI). Uncertainty is the word that describes this strategy, given the evident inconsistencies between the theory and the practice of the examination. Students underscore that the strategy in a way highlights their academic capabilities for the faculty to see; however, the opinion of the latter when the right answer is not provided is of concern. In this regard, Rosales¹² points out that, “teachers, in turn, play a new role in evaluations which is more similar to that of external examiners and moderators, given that they must control the process, protect students from unfair grades, and establish reference criteria for the evaluation.”

Students recognise that the aim of the test is to provide feedback, but they argue that this is often not done. Tobón¹³ states: “In the field of training, the essential goal of assessment is to provide feedback to students and faculty on how the competencies required for a given course or programme are developing.” Teachers recognise the challenges involved in preparing the SERI, but they express agreement with the strategy since they believe it gives more power to collective feedback. This is where the view by Rosales¹² is relevant: “Evaluation helps determine whether objectives have been attained or not, in order to go back on the things that were not learnt by the students, reinforce successes, and avoid repeating mistakes in the future.”

Objective Structured Clinical Examination (OSCE). The interaction between basic training and clinical practice becomes evident with each OSCE test, where the relationship between theory and practice is materialised in the real context category. The interaction is evidenced in simulated cases and real material such as laboratory tests or imaging studies, prompting the students to think about their patients from a more comprehensive and general perspective. Martínez¹⁴ states that, “any professional incentive and promotion system must consider these contents, and evaluation methods must be based on the job of the professionals in the real world.” Serdio¹⁵ points out that, “unlike other simpler and more direct tests, it is not just a matter of arriving at the right diagnosis or prescribing adequate treatment, but of practicing as good professionals in all instances, bringing to bear all their knowledge, skills and attitudes.” Martínez¹⁴ goes on to say, “the assessment of clinical competency is, therefore, an objective of institutions involved in training and using healthcare professionals.”

The test requires coordination on the part of the organisers. Serdio⁽¹⁵⁾ argues that “the logistics of this test involves a test committee as an essential component in an OSCE given that it is the collegiate body in charge of developing the content of test, consisting of a group of clinical professionals recognised for their experience.”

CONCLUSIONS

The study allowed to describe and understand innovative evaluation strategies used in the Quindio University Medical Programme: on-line examination, OSCE and SERI. All the strategies have their own strengths and weaknesses, although greater tension is found in association with the SERI. Students and faculty members look for ways to improve these strategies, and although students sometimes feel they are not heard by school authorities, they want to continue working on the improvement of all the strategies in order to achieve a comprehensive and satisfactory training.

RECOMMENDATIONS

Innovative evaluation strategies must be subjected to constant scrutiny in terms of structure and implementation, in order to strengthen comprehensive student training.

Work must be done to enhance the stability of the platform for the on-line examination in order to ensure full recording of the entire test and preempt manipulation by third parties. Training of the faculty in the management of virtual environments must also be encouraged.

The SERI is a strategy that must be reflected upon by the faculty in order to ensure it is a real space for feedback and does not become a threat to the self-esteem and knowledge assessment.

The OSCE is very well regarded by faculty and students alike. Faculty members need to work harder on the organisation of the test, in particular the layout of the stations. A larger physical space is also needed in order to allow students and faculty to move comfortably between stations.