Wrong answers for wrong reasons
the risks of ad hoc instruments
Additional information
Authors
Chiodini L.,
Hauswirth M.
Type
Article in conference proceedings
Year
2021
Language
English
Abstract
To evaluate novel pedagogies, approaches, and tools, Computer Science Education researchers often conduct experiments to look for differences among groups treated with different interventions. The methodological rigor of such experiments affects the soundness of the conclusions the researchers can draw. In this paper we focus on a central aspect of such experimental research: the instruments used to assess participants’ knowledge. Specifically, we study the use of ad hoc instruments and the risks due to their insufficient validation. We present a literature survey that highlights how, even though standardized instruments exist, the majority of published experiments in the last five years at major Computer Science Education conferences carries out pre/post-tests using ad hoc instruments, often with multiple-choice as question type. We demonstrate the risks of such commonly used but insufficiently validated multiple-choice instruments. We propose a richer way to analyze and assess the correctness of answers to multiple-choice questions, requiring participants to add brief explanation texts as a justification of each answer. We run an experiment and analyze the collected answers using the two approaches, with and without explanations, to show that the risk of drawing opposite conclusions from the statistical analysis is real.
Keywords
Computing education, Experiment, Assessment, Multiple-choice
Conference proceedings
Koli Calling International Conference on Computing Education Research
Numero ( Mese )
16
Meeting name
Koli Calling '21: 21st Koli Calling International Conference on Computing Education Research
Meeting place
Joensuu Finland
Meeting date
November 18, 2021
Pages (or article number)
1-11
Diffusion
License
Rights reserved
Visibility
Public
Status open access
Green
