Announcing
PhD in Second Language Studies Dissertation Defense
Patharaorn Patharakorn
Assessing interactional competence in a multiparty roleplay task: A mixed-methods study
Chair: James Dean Brown
Friday, November 30, 12:00 p.m.– 2:00 p.m.
Sakamaki Hall, B104
Abstract
In an effort to develop an assessment instrument in measuring interactional competence (IC) with a method that is congruent with the current research findings on IC and IC development (e.g., Hall, Hellermann, & Pekarek Doehler, 2011; Pekarek Doehler & Pochon-Berger, 2015), the present study investigated students’ performances on a multiparty roleplay on a task called Socializing. Using the sequential mixed methods design (Greene, 2007; Tashakkori & Teddlie, 2003), the study explored empirical evidence garnered through qualitative and quantitative research methods to test if the proposed rubric can provide a valid and reliable measurement of IC on this performance assessment task.
The participants of this study were 180 undergraduate engineering students at a university in Thailand who were taking an EFL course that targets social communication skills in professional contexts. Students were randomly grouped together and were asked to have a conversation for 10 minutes, in which they must introduce themselves as their character and try establishing business contacts for the companies. The data for this study included 34 video-recordings of the group roleplay performances.
Conversation analysis (Clift, 2016; Sacks, 1992; Schegloff, 2007; Sidnell & Stivers, 2013) was employed to identify comparable interactional activities and determine the interactional methods students utilized in carrying out those activities. The productive activities are self-introduction, work talk, business contact exchange, post-conference arrangement talk, and an interaction to bring about the termination of the roleplay performance. Three recipient actions include students’ management and display of their understanding, students’ management of alignment, and finally, their display of affiliative stance.
Six raters from various teaching and training backgrounds were recruited to apply the proposed rubric in evaluating the students’ IC on the eight items, combining both productive and recipient actions. The Many-Faceted Rasch Measurement (Linacre, 1989) with the Partial Credit Scoring model (Masters, 1982) provided integrated measurement reports of the rating practice. The findings revealed that students’ ability on this IC construct mostly exceed the difficulty of the Socializing task. Self-introduction and understanding display had been identified as the two easiest items, followed by alignment display, work talk, affiliation display, activity termination, making post-conference arrangements, and bringing up contact exchange, respectively. The analysis also suggested that most raters were reliable in applying the rating scale, though they demonstrated a higher degree of uniformity in evaluating productive activities compared to their ratings of recipient actions. Overall, the mixed methods research design is seen to have provided a much-needed framework in this process of exploring the validity evidence of the proposed rubric in assessing IC for the multiparty roleplay performances on the socializing task.
References
Clift, R. (2016). Conversation analysis. Cambridge: Cambridge University Press.
Greene, J. C. (2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.
Hall, J. K., Hellermann, J., & Pekarek Doehler, S. (2011). L2 interactional competence and development. Bristol: Multilingual Matters.
Linacre, J. M. (1989). Many-faceted Rasch measurement. Chicago, IL: MESA Press.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
Pekarek Doehler, S., & Pochon-Berger, E. (2015). The development pf L2 interactional competence: evidence from turn-taking organization, sequence organization, repair organization and preference organization. In T. Cadierno & S. W. Eskildsen (Eds.), Usage-based perspectives on second language learning. Berlin, Boston: De Gruyter Mouton.
Sacks, H. (1992). Lectures on conversation (Vol. 2). Oxford: Blackwell.
Schegloff, E. A. (2007). Sequence organization in interaction: A primer in conversation analysis. Cambridge: Cambridge University Press.
Sidnell, J., & Stivers, T. (Eds.). (2013). The handbook of conversation analysis. West Sussex, UK: Wiley-Blackwell.
Tashakkori, A., & Teddlie, C. (2003). The past and the future of mixed methods research: From data triangulation to mixed methods in social and behavioral research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research. Thousand Oaks, California: Sage.