Visit by Dr. Magdalena Wolska

On Thursday, April 3rd 2014, Dr. Magdalena Wolska will visit UKP to meet with researchers and give a talk.

Title: Clustering-based Computer-assisted Scoring of Short Constructed Responses: An Intrinsic and Task-based Evaluation in a DaF Placement Testing Scenario

Abstract: Computational analysis of short constructed responses, as elicited in educational testing, has been gaining increasing attention within the language processing community. Short answer scoring has already had its own SemEval task and a Kaggle competition. Beyond it being a non-trivial problem to tackle for computational semantics, the interest in the task is also rather practically motivated: automating the scoring process would decrease the overall cost of conducting assessments. While automated systems are becoming more sophisticated and accurate, it is not clear whether (or rather when) scoring accuracy acceptable for deployment in high-stakes testing can be reached.

In this talk, I will present our work on an alternative approach to short-answer scoring -- computer-/assisted/ scoring -- in which clustering is used to group similar responses for grading, with a view to streamlining the process and reducing the grading time. First, I will present a simulation study which assumes that a rater assigns a score to only one cluster element and show that based on this strategy, above 80% accuracy can be achieved by scoring (labelling) only 40% of our DaF listening comprehension data. In the second part of the talk, I will present an evaluation experiment which we conducted as part of /actual/ DaF course-placement testing at the Saarland University. Our results show that in a real-life setting, manual scoring of clustered responses proceeds significantly faster than scoring responses presented ordered by frequency of occurrence and as fast as scoring response sheets on by test-taker basis, the mode which is most familiar to our raters, teachers at the language centre.

Bio: Magdalena Wolska is a researcher at the Learning, Educational Achievement, and Life Course Development Graduate School of the University of Tuebingen. Before joining LEAD she was a researcher at the Saarland University, research programmer at the NLP group of the Educational Testing Service, and a consultant at the now defunct consulting firm, Kinderhook Systems, Inc. She defended her PhD in computational linguistics at the Saarland University, her MPhil, also in computational linguistics, at the University of Wolverhampton (UK), and studied in an Information Systems programme at the Wrocław University of Technology (Poland).