Search for contacts, projects,
courses and publications

Evaluating Recommender Systems and Beyond: Practical Application of Statistical Procedures

People

Di Penta M.

Course director

Description

Empirical evaluation has paramount importance in any field of scientific research, to assess an approach's performance, to relate variables from a phenomenon, and just to interpret observational data. Performing an appropriate empirical evaluation implies applying very rigorous criteria in defining the goals of a study, designing it, and, ultimately, analyzing and interpreting the results of a study.

Stemming from my personal experiences - for the specific case of evaluating recommender systems for software engineerings - I will first introduce the audience to the tricky path one has to follow in empirical evaluations. As I will discuss, performing an offline evaluation of the approach precision and recall is only a very preliminary starting point. Importantly, different kinds of (qualitative and quantitative) evaluations, having a different size and level of control, and above all involving humans, are required to achieve results able to convince practitioners of the actual usefulness and applicability of a tool.

Then, I will overview and explain, through simple examples as well as through cases from my previous research, the different kinds of statistical procedures to be applied in different cases, illustrating how they can be applied, and the typical mistakes one should avoid.

Final note: too often we are discouraged in conducting a very strong empirical evaluation because it would delay our research. I will illustrate how in many cases a relatively affordable initial investment is worthwhile in terms of what one can gain from it.

 

The seminar will be complemented with a session of (offline) practical exercises in which the participants will be able to apply what learned during the lecture. These sessions are mandatory for students who want to receive the ECTS assigned with this course (1 ECTS).

Education