Announcing
PhD in Second Language Studies Dissertation Defense
Gerriet Janssen
English Proficiency and Entrance to Doctoral Studies at One Colombian University: A Standard Setting Study
Chair: James Dean Brown
Tuesday April 26, 2:00 p.m.
Moore Hall, Room 155A
Critically important to test and program development is the enterprise of standard setting. This activity establishes descriptions of different performance levels and then uses a principled methodology to construct a cut score on a test: a point on a score scale that represents the separation of these different performance levels. To ensure that the creation of this cut score has been done is a way that is “systematic, reproducible, objective, and defensible” (Cizek & Bunch, 2007, p. 8), standard setting draws upon a great body of theoretical and empirical resources, protecting test users from the negative consequences of arbitrary decision making.
Recently, one Colombian university developed coursework and a placement exam for an EAP program supporting PhD student publication and presentation skills. During this program’s development, a classification exam was created to place the PhD students into different course levels. Though not part of this exam’s uses, university leaders soon established that certain course levels would now be an entrance requirement for different PhD programs. This decision was arbitrary in that it was made without understanding (a) what decision makers wanted their PhD students to be able to do in English or (b) how these skills intersected with the different course levels that were being offered. In short, this decision was made without taking advantage of the technology of standard setting.
Responding to this situation, this study—using the Body of Work Method (for the performance writing and speaking sections of the exam) and the Bookmark Method (for the multiple-choice grammar-vocabulary-reading comprehension section)—elicited from this university’s PhD community their conceptualization of the minimum English proficiencies for PhD study at this university. From these individual recommendations, one cut score was initially calculated; however, the different procedural, external, and internal validity evidence indicated that two cut scores—one more lenient, one more severe—would be a better solution for this context. However, in light of other evidence from the validity argument, this dissertation suggests as its final recommendation that the university prioritize which tools of learning it would like to develop, instead of worrying about which limitations it should impose.