Print version ISSN 1657-0790
profile vol.12 no.1 Bogotá Jan./June 2010
Designing Proficiency Tests to Accredit Previous
Knowledge in American and British Literature
in a Bilingual Education Program
Diseño de exámenes de suficiencia para acreditar el conocimiento previo
en literatura americana y británica en un programa de educación bilingüe
César Julio León Chica*
Catalina DCosta Martínez**
Gisela Franco Jácome***
Universidad El Bosque, Colombia
* E-mail: firstname.lastname@example.org
** E-mail: email@example.com
*** E-mail: firstname.lastname@example.org
This article was received on August 1, 2009 and accepted on January 20, 2010. This article aims at identifying the kind of American and British literature tests that can be designed to allow students who enter a bilingual education program at a private university in Colombia to have their previous knowledge in these two subjects accredited through a proficiency test. Students' needs, opinions, beliefs, existing commercial tests, the University's (specifically, the one where the study was conducted) literature programs, several anthologies, and competences required in the education program were all taken into consideration. Then the tests were developed, piloted, and validated with a focus group composed of ten students of said program. The results indicated that students require previous knowledge, literary competence, and command of the English language because those are determining factors in successfully passing the tests. Key words: Previous knowledge, proficiency tests, American and British literature, competences
This article aims at identifying the kind of American and British literature tests that can be designed to allow students who enter a bilingual education program at a private university in Colombia to have their previous knowledge in these two subjects accredited through a proficiency test. Students' needs, opinions, beliefs, existing commercial tests, the University's (specifically, the one where the study was conducted) literature programs, several anthologies, and competences required in the education program were all taken into consideration. Then the tests were developed, piloted, and validated with a focus group composed of ten students of said program. The results indicated that students require previous knowledge, literary competence, and command of the English language because those are determining factors in successfully passing the tests.
Key words: Previous knowledge, proficiency tests, American and British literature, competences
Este artículo busca identificar un modelo óptimo para un examen de literatura americana y británica que se puede diseñar para que los estudiantes que inician el programa de educación bilingüe de una universidad privada en Colombia puedan acreditar el conocimiento previo en estas dos materias, mediante un examen de suficiencia. Para el diseño de los exámenes se tuvieron en cuenta las necesidades y opiniones de los estudiantes, así como exámenes comerciales existentes, los programas de literatura de la universidad, varias antologías y las competencias requeridas por el programa de educación. Los exámenes diseñados se pilotearon y se validaron con un grupo de 10 estudiantes del programa en mención. Se determinó que los estudiantes requieren un conocimiento previo, competencia en literatura y un nivel competitivo en inglés, para aprobar satisfactoriamente los exámenes.
Palabras clave: conocimiento previo, acreditación, exámenes de literatura americana y británica, competencias
Literature has always been one of the strongest interests in the study of bilingual education. This leads to the focus of this research study which aims to find out what previous knowledge in American and British literature undergraduate students in the bilingual education teaching program bring with them to the university. The question of which process would be appropriate for the accreditation of this previous knowledge arose. According to Webster's Third New International Dictionary (1986, p. 13), "Accredit is, 1. to put (as by common consent) into a reputable or outstanding category: consider, recognize, or acclaim as rightfully possessing uncontested status; 2. to give official authorization to or approval of ". On the other hand, accreditation is defined as "the act or process of accrediting" which means that the knowledge students already had when they entered a university program could be validated in either of two possible ways described in the university's student handbook: The first, homologation, is the equivalent of transfer credits and gives the same amount of credits for a subject studied at a different university or institution (Ministerio de Educación Nacional [MEN], Decree 0808 April 25, 2002, Articles 3, 4 & 8). The second is a proficiency exam, which allows the student to prove his/her knowledge in a certain area by taking an exam. This has its legal foundations in Law 30 (Ley 30), Articles 20 & 28, authorizing each university to develop its own student handbook where it specifies how it is going to handle different academic aspects (Colombian Congress, 1992).
The interest in this particular topic arose from our experience as English teachers and some of the Education students who also possessed a background of previous knowledge in both American and British literature. Since they could not get their previous knowledge accredited, it was proposed that the University let us design the two proficiency tests to be used as a tool to accredit previous knowledge in those subjects.
The researchers' goals were to open doors in the Faculty of Education of the University that would allow students' previous knowledge to be taken into account and allow future generations to be able to sit exams on the subjects which they considered themselves proficient in. Accreditation saves students valuable time and, therefore, enables them to study new subject areas, thus the students of the Bilingual Education Program at the University will optimize their time and their knowledge by getting the credits in these subjects accepted. When adults are studying and working full time, besides being parents and having other family obligations– as is the case of most of the students at this university– time is precious.
The Research Questions
Our concern about this topic has evolved, and in order to take appropriate action, it was decided we should investigate this subject in depth. After having studied the problem, the following research question was posed: How can the University's bilingual education students' previous knowledge, the existing literature proficiency tests, and the University's current American and British literature programs be taken into consideration in designing proficiency tests for these two subjects? Hence, we formulated the following specific questions to find a solution to said problem: What previous knowledge of American and British literature do students bring to the university? What type of tests can be designed to evaluate students' previous competence in American and British literature? How can the criteria of the topics be tested and developed?
Imagine that instead of reading literature, you
heard it told to you by storytellers around a
campfire or a fireplace. Telling and listening to
make up stories, rather than writing them down,
is part of the oral tradition. Eventually, versions of
these tales were written down.
(Carroll, Feldman, Kinsella,
Stump & Wilson, 2002, p. 10)
Folk tales travel wherever people travel.
(Anaya, 2002, p. 10)
After establishing the legal foundations to accredit the previous knowledge through the proficiency tests, we focused on defining the four important constructs of our research: proficiency tests, literature, prior knowledge, and competence.
According to Hughes (1989), "proficiency tests are designed to measure people's ability in a subject regardless of any training they may have had in that subject. The content of a proficiency test, therefore, is not based on the content or objectives of any courses which people taking the test may have followed" (p. 9). If this is so, then theoretically anybody who has ability to perform in a specific subject is entitled to take a test to prove their abilities. In this case, knowledge and previous knowledge are not taken into account. "Rather, proficiency is based on a specification of what candidates have to be able to do in order to be considered proficient" (Hughes, 1989, p. 9). In Webster's Third New International Dictionary (1986), the definition of proficient is the following: "to be well advanced in an art, occupation, skill or a branch of knowledge: unusually efficient".
One of the proficiency tests for literature recog-nized worldwide is the Graduate Record Exa-mination (GRE) subject test (literature in English), which is not for undergraduate students, of course. Even though it is a useful tool for comparative purposes, the researchers decided that the Subject Area Test (SAT) literature tests recommended by College Level Examination Program (CLEP) are more adequate to the level and needs of the participants and the University.
Regarding the abovementioned data, Hilliard (1999) asserts that "All standardized tests have their place, even the weak ones. Without them we would have nothing —no way to hold people accountable for teaching or learning" (p. 38). In connection with this, our goal was to design the proficiency tests in order to account for students' competences in literature through the mentioned tests. Regarding tests, Hilliard (1999, p. 63) asks: "Then why use standardized tests at all?" Schools and districts use them because they provide data and results that are essential for improvement. For these reasons, we took the theory and some models as test references.
Student performance data are essential elements in a healthy and successful improvement effort (Glickman, 1993; Fullan, 1991). These authors state that standardized tests, despite some shortcomings, provide numerical and intelligible data as to how well a learner is performing or improving, and vital information about patterns of strengths and weaknesses among students.
To define this construct (a subject that focuses on special topics because it is content-based), we can start with Long (2005), who says that literature is the written record of man's best thoughts and feelings, and English literature is the part of that record which belongs to the English people. In its broadest sense, literature includes all writing, but as we commonly define the term, it excludes works which aim at instruction and includes only the works which aim to give pleasure, and which are artistic in that they reflect nature or human life in a way to arouse our sense of beauty (Long, 2005).
According to the Encyclopedia Britannica (1994, p. 398), literature is a body of written works. The name is usually applied to those imaginative works of poetry and prose distinguished by the intentions of their authors and the excellence of their execution. Literature may be classified into a variety of categories, including language, national origin, historical period, genre, and subject matter. When we study literature we deal chiefly with the great, enduring books, which may have been written in an older or a recent time, but which have in them timeless magic. In order to be considered proficient in literature, a person must understand each form of writing and be able to identify each text in its corresponding genre.
As already said, this investigation intends to determine the role prior knowledge plays in determining what type of test will be designed in order for students to be acknowledged proficient in American and British literature.
According to González (2000), "what we don't know prevents us from learning". The author says that many research studies uphold the importance of a reader's prior or previous knowledge. It is a fundamental variable in reading comprehension and in learning from texts.
The researchers agree with him, and think that this is especially applicable when referring to literature. Besides, González (2000) states that there are special tasks focused on activating previous knowledge. In connection with this concept, Hitotuzi (2005) mentions that "Students are not a tabula rasa on which a supercilious teacher can record their knowledge; instead, they have knowledge and experiences of life and language (as well as other subjects, contents, and topics) which can contribute greatly to the learning process" (p. 98). In our case, some students of the Bilingual Education Program had some previous knowledge and experience in literature and thus wanted this knowledge to be accredited through a proficiency test.
Hughes (1989) states that prior knowledge and content seem to have a certain degree of influence regarding proficiency; thus, he supports the idea that if students have previous knowledge of a topic –in our case, literature– it will affect their test results. Similarly and following Roschelle (2004), it can be stated that it would be very difficult to learn without prior knowledge because students construct new concepts from prior knowledge. Since prior knowledge is based on experiences, it can be said that new knowledge is also defined by the new experiences in life.
Rico (2005) adds that "The level of proficiency is seen according to the students' capacity to recognize and master the elements of the system" (p. 99). Furthermore, he links several important concepts related with test design: proficiency, test design, competence, and previous knowledge. These elements supported our main goals in designing the proficiency tests in which, first, we found out what previous knowledge of American and British literature the participants had. In doing so, we followed Hughes (1989), who emphatically points out that the first step in test designing or construction is the statement of the problem. The test must be "perfectly clear about what it is one wants to know and for what purpose" (p. 48).
According to the Colombian National Ministry of Education, competences are a set of knowledge, attitudes, disposition and skills (cognitive, socialaffective and communicative), all interrelated to help students in the learning process, including the development of a new sense of activity in new and challenging contexts (MEN, 2004). Therefore, competence implies knowing, being, and knowing how to do. From the above, competences can be seen as the battery of knowledge, skills, and attitudes needed to carry out a determined activity and/or to solve problems in an autonomous and creative manner.
According to Smith & Wilson (1990), "Within Chomskyan theory, there are two levels of assessments: the first is called the level of competence, the speaker's knowledge of language which is purely linguistic, and the second is called the level of performance, the speaker's use of language" (p. 20).
Competences play an important role in the program under study. When a student wants to accredit previous knowledge in literature, s/he should not only demonstrate competence in the English language, but in the literature subject. He needs to perform and accomplish tasks in regards to aspects such as the ones mentioned before: national origin, historical period, genres, and outstanding works, among other aspects of the subject matter.
Type of Study
The study was a descriptive qualitative research because "the researchers do not set out to test hypotheses, but rather to observe what is present with their focus, and consequently the data, free to vary during the course of the observation" (Larsen-Freeman & Long, 1993, p. 10). According to these authors, this study follows a cross-sectional approach where the performance of a large group of subjects is studied, and the data are usually collected in one session. Hence, we gathered the data through personal answers of the interviewees by means of questionnaires and oral interviews.
According to the above-mentioned authors, "one of the characteristics of the qualitative paradigm is that it is not generalizable; it is process-oriented, subjective, grounded, discoveryoriented, exploratory, descriptive, and inductive. It is also holistic and assumes a dynamic reality" (Larsen-Freeman & Long, 1993, p. 11). First of all, we wanted to find out if the participants had previous knowledge, and if so, if they were willing to take the tests to accredit their previous knowledge, to then be able to design the literature tests.
The Participants and Setting
The study took place at a private university in Bogotá. The undergraduate program is called B. Ed. Program in Bilingual Education with Emphasis on Teaching English (Licenciatura en educación bilingüe con énfasis en la enseñanza de inglés). It is one of the five options offered by the Faculty of Education. At the time that it first opened, it was the only Faculty of Education in Colombia which offered a program specifically in bilingual education. This program has an afternoon schedule which leads us to think all its students are not recently graduated from high school. Part of the population registered for this program is comprised of teachers who have many years of experience and want to increase their professional standing. Others are young teachers who need to work while studying for their degree.
Out of sixty four students of the Faculty of Education, ten students were selected as a focus population to answer a prior information questionnaire (See Appendix 1). They replied that they did have previous knowledge in American and/or British literature, and also answered affirmatively to the question of wanting to present a proficiency test to have this knowledge accredited. Their ages ranged from sixteen to fifty five. Some of them studied at bilingual schools in the city; others either studied or lived for a time (varying with each individual) in English speaking countries. Other students gave account of their previous knowledge in these areas due to personal interest and autonomous study.
Instruments and Procedure
As already said, at first we developed a Prior Information Questionnaire (See Appendix 1). We selected a questionnaire to collect the first information because it let us gather data in one or two sections. As Burns asserts, questionnaires are "written sets of questions used to gain responses in non face-to-face situations; questions are usually focused on specific issues and may invite either factual or attitudinal responses" (1995, p. 117).
After selecting a focus population, an interview was designed by the research team to ask the target population their opinions, beliefs and experiences with previous proficiency tests (See Appendix 2). The information was obtained by directly asking the chosen and consenting participants, and recording their answers with an audio tape recorder; it was then transcribed in the computer for analysis. We bore in mind Siedman (1991), who, concerning interviewing as qualitative research, states that "the purpose of in-depth interviewing is not to get answers to questions, nor to test hypotheses, and not to 'evaluate' as the term is normally used; at the root of in-depth interviewing is an interest in understanding the experience of other people and the meaning they make of that experience" (p. 19).
Those two instruments were chosen after reflecting on the type of question to be answered and the type of research that was being carried out. The information was typed into the computer, then organized; lastly the researchers carried out the analysis by closely following the steps that dealt with levels of analysis.
In the Literature Review section it is stated that one of the main limitations of proficiency exams or standardized tests is the fact that they are designed to be taken by large groups of people at one time and, therefore, must necessarily be machine scored (Barr, Craig, Fisette & Syverson, 1999; Hilliard, 1999; Covey, 1989; ETS, 2002). Thus, commercial standardized tests were not a good option for our purpose. To avoid this limitation, the researchers took it upon themselves to design two "tailor-made" literature proficiency tests. In this task we considered the students' needs, opinions, and beliefs, as well as the Faculty's requirements for the above-mentioned subjects. Then the tests were developed and piloted with the previously chosen candidates.
In the designing of the tests we took into consideration the data provided by the participants through instruments described above, as well as information from several literature textbooks, anthologies, the GRE, the CLEP, and the SAT. All these documents are related with literature subject tests and were used as references to design the two literature tests.
The participants took the tests. Then, both tests were assessed (graded) by the researchers and a final questionnaire was presented to the participants in order to learn their opinions about the literature tests (See Appendix 3).
After the ten interviews were carried out, they were transcribed uniformly in order to be analyzed with the computer software Atlasti. This program is a tool designed with the objective of facilitating the qualitative analysis of large quantities or volumes of data, or information which has been gathered for research purposes. It is not meant to replace the human interpretation of the analyzing process, but its main purpose is to help to accelerate many of the activities implied in the qualitative analysis and interpretation such as text segmenting, quoting, and coding.
Students' Opinions Regarding the Type of Test That Should Be Designed
The use of this program allowed the researchers to define certain categories based on the common patterns that resulted from the analysis of data. These patterns provided both quantitative and qualitative information about the students' opinions regarding the type of test that should be designed to evaluate or to accredit their previous knowledge in literature. The categories found were previous knowledge, reasons for taking the test, type of test, number of questions, and testing time, as can be seen in Table 1. These categories were defined by taking into consideration the number of times the target words were mentioned by each interviewee when we used the format shown in Appendix 2.
The categories shown in Table 1 had, in turn, several subcategories. With the purpose of designing the literature proficiency tests, the researchers had to find out what the students thought the focus of proficiency test should be. There were three questions in the interview that focused on these aspects: One theory of the two literature tests previously mentioned, CLEP and SAT, considers that a well prepared examination may include the following: authors, texts, historical periods, genres and figurative language, among others. The students' answers reflect the same, since the interviewees said that they would like tests about authors, topics, historical contexts, different genres, and a combination of the above.
'Reasons' was an important category for us, due to the need to know why the students wanted or were willing to take a proficiency test. The main motivations observed were to save time and to advance subjects in their studies. It should be noted that when the surveys and interviews were analyzed, the answers confirmed that the existing reasons stated above for taking proficiency exams were widely shared by the students at the University. The Faculty's Dean and Academic Council understood these reasons and let us carry on with the design and validation of the tests.
'The students' interviews regarding type of questions revealed that they did not want closed questions based on memory. They expressed that they wanted short, open-ended questions, essays, and reading comprehension.
The majority of the students interviewed answered that the number of questions should be between 1 and 20. As to the number of questions the test should have, 8 students said that they wanted from 1 to 20 questions; one student said that it should have from 1 to 10 questions; three students said from 1 to 50, and one student said from 1 to 100. However, taking into account the extent of the topics to be examined, and comparing it with the number of questions of the existing standardized exams, we considered that this was a very low number, as the average number of questions in existing literature proficiency examinations for undergraduate students is about one hundred. The GRE has 230 questions, but this is only a reference source as it is an exam for students who have completed a degree in literature and are seeking to obtain a master's degree. This did not comply with our case, which was proficiency exams for undergraduate (education, not literature) students. Therefore, we had to focus on the SATCLEP examinations for accrediting literature as undergraduates, which is the nearest equivalent to what we were expecting to achieve in our program. Considering the scope of the University's currently existing programs in American and British literature, equivalent tests, and students' answers, we considered that the adequate number of questions should be around 60 or 70 because not all of them were going to be multiple-choice and we would include some short answer, openended questions, and one essay question.
The category testing time included two subcategories, and was aimed at finding out students' preferences as to when, during their studies, they would want to take the proficiency tests. Of the total of answers given, the majority indicated that they would take the test any semester; and the others indicated that they would like the opportunity of taking it during the first semesters (1- 3) because what they had learned at school was still fresh in their minds. This question was solely for informational purposes because it had no usability. As noted by Brown, it "has to do with the degree to which a procedure is practical to use". However, the university students' handbook states that proficiency examinations may be taken at any time during the undergraduate studies (1995, p. 52).
Taking into account that the objective of the study was to find out what the role of the previous knowledge was for designing a literature proficiency test, this was one of the most important categories. Questions one, two, and eight from the interview provided the greatest amount of information regarding this category (See Appendix 2). Table 2 illustrates this, with answers given by three participants, about previous knowledge.
The students' comments in Table 2 supported the quote from the CLEP (2007) in which it states that only if they had this "broad knowledge of literature" and "a familiarity with the basic literature terminology", the participants were able to obtain good results as was observed when the tests were applied.
The Designing of the Proficiency Test and Its Outcomes
Having analyzed all of the above information from our three sources, students' interviews, theory (University's current programs, standardized tests, and anthologies), and the researchers' points of view, we decided that the tests in American and British Literature should have 70 questions each and one essay. These questions were thirty multiple-choice (30 points), a time line (10), reading comprehension (20), ten open-ended questions (10), and one essay (30). The numbers in parentheses are equivalent to the number of points each type of question would count for a total of one hundred points (100). The minimum passing grade was 70/100.
The authors were chosen from those included in the current University's programs for each of the subjects, and the most well-known authors not included in these programs, but who were taken from high school literature anthologies.
After having corrected the literature proficiency tests presented by ten students at the University, the general results were as follows: half of the total number of students who took the literature proficiency tests passed them- a majority passing British Literature and a minority passing American Literature.
In British Literature a great number passed the time line, multiple-choice, and reading hension; less than half passed short opened-questions; and everybody did well on the essay. Finally, all of the students achieved a passing grade on the test, (60/100), but since the minimum grade for obtaining the proficiency credits was placed at 70%, most of the participants achieved it, but one person did not. The time allotted was adequate because all of the students used fewer than the three hours given, and only one person took two hours and fifty minutes.
In American Literature nobody passed the time line; one person passed the multiple choice section; one person passed the short open-ended question; and several passed the essay. One person did not complete it, and one person did not do it because she felt unable to do it. One person obtained a perfect score (30/30) on the essay.
The time allotted was enough because all of the people who took the test needed less time to complete it. After the students sat the literature proficiency tests, they were asked to answer a final questionnaire (See Appendix 3) in order for the researchers to obtain feedback on the tests. Some aspects worth mentioning about the feedback results are the following: Six students did not study at all for the test. However, five of them passed the test, which supports the idea that previous knowledge is important when taking a proficiency test. Participant 5 said: "No studying; I just relied on previous knowledge".
Before the students knew their results, more than half thought they had done acceptably and the rest thought they had done poorly. The main reasons mentioned were lack of study and preparation. Nevertheless, the tests results obtained were usually better than what the participants thought. For example, participant No. 2, who answered acceptably, did very well (93/100); likewise, participant No. 9, who answered poorly, obtained the passing grade of 70/100.
Regarding the allotted time, three participants expressed that it was sufficient; four said it was adequate, and two thought it was excessive. In general, the students' acceptance of the test was good even though they suggested some modifications. This can also be observed in the explanations given to question No. 7. Some of these explanations contradict each other. As none of the reasons given was supported by a considerable majority, the researchers concluded that the tests should not be modified.
Findings, Implications, and Applications
The tests were based on the type of questions that students wanted to have included as well as what the researchers determined from studying the university's programs, the existing commercial tests, and several literature anthologies. This made it possible to combine both proficiency and achievement (theory-practice; previous knowledgeability/ competence) in a more communicative type test. In general the researchers thought that the most difficult part of the test was going to be the essay, but the students' results show that all of the participants did well, even those who expressed that it was difficult and long.
As researchers, we thought that the participants were going to do very well on the reading comprehension part, as the answers can all be found in the text given if the participant has good reading skills, and because in the interview eight out of ten wanted it to be included in the test. However, this section did not produce the expected results. This may be due to the fact that many students read too fast, and did not pay sufficient attention to details during testing.
The above has a definite relationship with what Glickman (1993) expresses in that proficiency tests are useful in determining patterns of students' strengths and weaknesses in subject areas.
With respect to the short open-ended questions, the results obtained by many of the participants were poor. This was unexpected because the questions included in the tests were designed around very well-known topics. It was also strange because in the interview students answered (see Appendix 2) about the type of questions to be included in the test, three of the participants specifically asked to include this type of questions.
It is important to acknowledge the fact that the profile needed for presenting these tests was that of having prior knowledge in either American or British literature, or in both. Seven of the participants met this requirement either because they studied at bilingual schools or because they were teaching literature. This issue brings out the fact that there were three students who presented the proficiency tests without having the required profile.
Taking into account the students' feedback on the tests, the following aspects should be noted: "It should be based on the University program" (Participant 1); "Some authors are not in the program" (Participant 10). In connection to this, it should be kept in mind that the focus of a proficiency test is to validate the student's previous knowledge in a given subject in order to accredit it; therefore, it must include a general overview of the subject area.
A proficiency test is not intended to be limited to any course, curriculum, or single skill in the language. The following testimonies evidence some participants' positive views about the tests: "It's OK, better than the program" (Participant 2); "It was good, adequate to the topic" (Participant 4).
Although the test was the result of analyzing carefully the number of questions, the time allotted and the topics of several literature commercial tests, other general comments were that it was "too long" (Participants 7, 8, 9, and 10).
All the participants who passed with 70% or more had previous knowledge in literature from other studies either at high school, university or both, as was confirmed by the information obtained from the initial interviews. This finding was in addition to having taken the British literature course at the University in the semester just prior to taking the test. Another reason that influenced the above-mentioned difference was that all the students who took this exam and passed it had a high level of English and could be considered bilingual since they were taking or had already passed English VI at the University. This was a very important factor because the test requires a full command of the English language.
On the other hand, none of the participants had taken an American literature course at the University. Additionally, the three participants with the lowest scores on the test were still doing the first levels of English at the University at the time they took the test. It was inferred, therefore, that they are not totally proficient in the English language which, as stated above, is a determining factor in the final result. This is illustrated with this excerpt: "I did not even try to do the essay because I did not understand what it was asking" (Participant 4).
The experience learnt from this research project shows that previous knowledge must be validated in order to both let students save time in completing their career and having the possibility of studying other subjects. This can be supported with González's words (2000) when he expresses that many research studies uphold the importance of a reader's prior or previous knowledge. It helps, empowers, and is a fundamental variable in reading comprehension and learning from texts.
There is a definite relation between students' profile (background) and previous knowledge in order to obtain the accreditation of the latter, which is important to have in mind when the students ask to have their previous knowledge validated by presenting proficiency tests. If they do not have a good mastery of English, they will probably not be able to answer the essay.
As the majority of the students who took British Literature passed the test, the researchers think that the test may be a useful tool for the University in the future. Although the results were different for the America Literature test, it can be explained through the observations stated above that the reasons for this could be attributed to the students' lack of previous knowledge, and their low command of English, and not to failures in the test design itself.
This research demonstrated that the three aspects mentioned in the initial research question, students' previous knowledge, the existing literature proficiency tests, and the University's current American and British literature programs, were carefully studied and included in the design of the two proficiency tests. Our general objective was also achieved despite what Cohen (1994) says about standardized tests. This author believes there is a difference between proficiency tests, which are used mainly for academic purposes because they focus on theory, and achievement tests, used mainly for administrative purposes because they focus on abilities or competences.
Conclusions, Limitations, and Questions for Further Study
The main conclusion after having carried out this research project was that both previous knowledge and competences are determining factors in the test results. The difference in the test results among the participants who had previous knowledge both in literature as well as mastery of the second language supports what the theories and studies mentioned before state, regarding testing.
Another important finding was that competences, evaluated through reading comprehension and essay writing, can also be taken into consideration when assessing both previous knowledge and competence in literature.
Two literature proficiency tests were designed and validated although, according to TESOL literature, teachers do little collaborative research and most of the literature regarding proficiency tests is limited to language proficiency testing. The tests designed in this study achieved a synthesis of assessment methods, as well as a more communicative approach to literature testing. As proposed by Hilliard (1999), they evaluate previous knowledge (theoretical subject content/literature) in the timeline, multiple choice section, and short open ended questions, as well as competences considered in the reading comprehension and essay writing.
Some limitations we had to deal with in this research project were that ten people comprised a small population compared to the total number of bilingual education students (111 at the time this research was initiated). Nonetheless, this focus group was composed of the participants who met the required profile out of 64 participants who answered the prior information questionnaire.
When people volunteer to participate in a study like this, they do not understand the importance of their commitment to the successful carrying out of a project. This leads to the fact that complying with their responsibility was not the participants' priority, as it was for the researchers who depended on participants' willingness and time availability to answer and pilot the tests, and had to accommodate our schedule to theirs. Moreover, some of the interviewees' personal conditions made us fall behind schedule in scoring the tests, in classifying and analyzing the data. On more than one occasion, the researchers had to deal with interviewees' personal difficulties and problems. Another issue was that the only time that interviewees had was from 4 to 8 p.m. since most of them worked during the day and only arrived after 4:00 p.m. at the University.
Due to the lack of model literature tests that covered most of the aspects students wanted to be evaluated, we had to spend more time in designing them. Due to the nature of the tests developed for this project, they had to be teacher corrected and could not be machine scored.
After concluding the research we have the following questions for further study: How can the information obtained from the participants' interviews be used to modify or improve the University's current literature programs? How can the two proficiency tests resulting from this research be actually used by the University? Should they be revised or modified; if so, by whom? Should a question bank be developed in order not to repeat the exact test every semester? How can students' abilities and competences be taken into consideration to improve the University's literature programs? Can there be another way of accrediting previous knowledge?
The American and British literature tests were designed based on the students' needs, the University requirements and program, theory about literature and testing, and commercial tests to accredit previous knowledge. Competences and good command in English were essential factors in passing the tests. The tests were not published in this research to enhance its validity as they are going to be used by the University to accredit students' previous knowledge.
Accredit. (1986). In Webster's Third New International Dictionary, Inc., 1, 13. Proficient, 2, 1811. Chicago: Merriam- Webster. [ Links ]
Anaya, R. (2002). Folk literature. Timeless Voices, Timeless Themes, Copper Level. New Jersey: Pearson Education, Inc. [ Links ]
Barr, M. A., Craig, D. A., Fisette, D., & Syverson, M. A., (1999). Assessing literacy with the learning record. Portsmouth, NH: Heinemann. [ Links ]
Brown, D. H. (1995). The elements of language curriculum. Boston: Heinle & Heinle. [ Links ]
Burns, A. (1995). Qualitative research studies. Cambridge: Cambridge University Press. [ Links ]
Carroll, J. A., Feldman, K., Kinsella, K., Stump, C. S., & Wilson, E. (2002). Learn about literature. New Jersey: Pearson Education, Inc. [ Links ]
Cohen, A. D. (1994). Assessing language ability in the classroom. (2nd ed.). Boston: Newbury House. [ Links ]
Colombia. Ministerio de Educación Nacional [MEN]. (2002). Decree 0808 April 25, 2002 Articles 3, 4, and 8: Por el cual se establece el credito académico como mecanismo de evaluación de calidad de transferencia Art. 3, 4, 8. Bogotá. [ Links ]
Colombia. Ministerio de Educación Nacional [MEN]. (2004). ¿Qué son las competencias? Retrieved August 12, 07 from http://www.inesup.com/descargas/conceptodecompetencia.ppt [ Links ]
Colombian Congress. (1992). Ley 30 de 1992 art. 20, 28, 109: Fundamentos de la Educación Superior. Bogotá: Autor. [ Links ]
College-Level Examination Program [CLEP] (2007). Retrieved August 7, 2007 from http://www.collegeboard.com/ student/testing/clep/ex_el.html [ Links ]
Covey, S. (1989). The seven habits of highly effective people. New York: Simon & Schuster in Educational Leadership. [ Links ]
Esposito, J. J. (1994). Encyclopedia Britannica. Chicago: Encyclopedia Britannica, Inc. 7, 398. [ Links ]
Fullan, M. G. (1991). The new meaning of educational change. New York: Teachers College Press. [ Links ]
Glickman, C. D. (1993). Renewing America's school: A guide for school base actions. San Francisco: Jossey-Bass. [ Links ]
González, A. (2000). Lo que no sabemos nos impide aprender. Conocimiento previo y aprendizaje de textos [electronic version]. Universidad de La Rioja, 2001- 2007. Retrieved August 7, 2007 from http://dialnet.unirioja.es/servlet/articulo?codigo=1032524 [ Links ]
Hilliard, A. (1999). Toward a better teaching education: A conversation with Asa Hilliard. New York: Teachers Collage Press. [ Links ]
Hitotuzi, N. (2005). Teacher talking time in the EFL classroom. PROFILE, Issues in Teachers' Professional Development, 6, 97-106. [ Links ]
Hughes, A. (1989). Testing for language teachers. Cambridge: Cambridge University Press. [ Links ]
Larsen-Freeman, D., & Long, M. (1993). An introduction to second language acquisition research. New York: Longman. [ Links ]
Long, W. J. (2005). Outlines of English and American literature. An introduction to the chief writers of England and America, to the books they wrote, and to the times in which they lived. Retrieved August 23, 07 from http://www.fullbooks.com/Outlines-of-Englishand-American-Literature1.html [ Links ]
Popham, W. J. (1999). Why standardized tests don't measure educational quality. New York: Teachers Collage Press. [ Links ]
Rico, C. (2005). Searching for coherence in language teaching; the issue of teaching competencies. Colombian Applied Linguistics Journal, 7, 95-107. [ Links ]
Roschelle, J. (2004). Learning in interactive environments: Prior knowledge and new experience. Dartmouth: University of Massachusetts. [ Links ]
Siedman, I. (1991). Interviewing as qualitative research: A guide for researchers in education and the social sciences. New York: Teachers College Press. [ Links ]
Smith, N., & Wilson, D. (1990). Modern Linguistics. England: Penguin books. [ Links ]
About the Authors
César Julio León Chica holds a B.Ed. in Bilingual Education from Universidad El Bosque. He worked as an English teacher at the same university and currently works as a private English teacher. He is also the author of two English books.
Catalina D'Costa Martínez holds a B.Ed. in Bilingual Education from Universidad El Bosque. She works as a teacher in some subject matters at the same university and is also an official translator.
Gisela Franco Jácome holds a B.Ed. in Bilingual Education from Universidad El Bosque. She currently works as an independent English teacher.
The objective of this questionnaire is to establish which students, if any, of those studying Bilingual Education at the University, have any prior knowledge in the subjects of American and British Literature. If they do have this prior knowledge, we want to find out where and when they obtained it; and to ask if they want this knowledge to be accredited through a proficiency test.
If you are interested, it would be very valuable for our research project, if you would answer the following questions:
1. Do you have any previous knowledge on American Literature? Yes___ No ___
2. How would you consider your level in this area?
High = able to identify authors, writings, literary periods, and historical context and also to do reading comprehension activities about them. High? ____
Medium = able to identify most authors, writings, literary periods and do some reading comprehension activities about them. Medium? _____
Low = only identify a few authors and their work, but are not sure about literary periods or historical context. Not totally sure about doing reading comprehension activities. Low? _____
3. Where did you obtain this knowledge? School? ___ University? ___ Other? ___
4. When did you obtain this knowledge? Approximate Year? ______
5. Do you have any previous knowledge on British Literature? Yes___ No___
6. How would you consider your level in this area? *Use the same criteria as in question 2
7. Where did you obtain this knowledge? School? School? ___ University? ___ Other? ____
8. When did you obtain this knowledge? Approximate Year? _____
9. Would you consider taking a proficiency exam to validate this knowledge? Yes___ No ___
Taking into account the answers you gave in the prior information questionnaire regarding
literature proficiency tests that you answered on April 10 and 23, 2007, we would be very grateful if you would reply the following questions:
1. Would you like to take the proficiency test for American Literature? Why?
2. Would you like to take the proficiency test for British Literature? Why?
3. Would you like to take it/them during the first semester? Why?
4. If you have already started the program of Bilingual Education, when would you like to take it/
5. What kind of test would you like to take:
– Short Open-ended questions
– Closed questions e.g. fill in the blanks, multiple choice or true and false.
– Reading comprehension
– Essay type
– Other? _____________________ Which one ______________
6. Do you think that the test should be focused on?
– Historical Periods
– Different genres of texts
– A combination of the above
– Other_____ Which one ______________________________
7. How many questions do you consider that this test should have? From:
– Other? ______ Why? ________________________________________
8. What do you remember about your previous literature studies?
Independently of the results you achieved on the proficiency literature test you took, we would appreciate if you would answer the following questions truthfully and objectively.
1. Did you study for the test?
b. A little
c. More or less
2. How do you think you did on the test?
3. What was your mental and physical disposition at the moment of taking the test?
a. Excellent, healthy, rested, optimistic
b. Tired, not well, sick, negative
c. Other? Explain ___________________________________________________________________________
4. How did you feel when you answered the literature proficiency test?
5. Did you think the test was?
6. Regarding the allotted time (3 hours) was it?
a. Not enough
7. About the topics and questions, did the test:
a. Meet your expectations?
b. Was it too general?
c. Was it too specific? Explain your answer.
8. Grade the type of question from 1 to 5 being:
1 = very easy, 2 = easy, 3 = average, 4 = difficult, and 5 = very difficult.
9. Observations: in your own words give us any feedback or suggestions to improve the test.
After knowing the results you obtained in the proficiency test, please answer the following two questions:
10. Grade the type of question from 1 to 5 being:
1 = very easy, 2 = easy, 3 = average, 4 = difficult and 5 = very difficult:
In your opinion, what part of the test would you modify?