Fixed-cost Pooling Strategies based on IR Evaluation Measures

Authors: 
Aldo Lipani
João Palotti
Mihai Lupu
Florina Piroi
Type: 
Proceedings contribution
Proceedings: 
ECIR 2017: Advances in Information Retrieval
Publisher: 
Springer, Cham
Pages: 
357 - 368
Year: 
2017
ISBN: 
ISBN: 978-3-319-56608-5
Abstract: 
Recent studies have reconsidered the way we operationalise the pooling method, by considering the practical limitations often encountered by test collection builders. The biggest constraint is often the budget available for relevance assessments and the question is how best - in terms of the lowest pool bias - to select the documents to be assessed given a fixed budget. Here, we explore a series of 3 new pooling strategies introduced in this paper against 3 existing ones and a baseline. We show that there are significant differences depending on the evaluation measure ultimately used to assess the runs. We conclude that adaptive strategies are always best, but in their absence, for top-heavy evaluation measures we can continue to use the baseline, while for P@100 we should use any of the other non-adaptive strategies.
TU Focus: 
Information and Communication Technology
Reference: 

A. Lipani, J. Palotti, M. Lupu, F. Piroi, G. Zuccon, A. Hanbury:
"Fixed-cost Pooling Strategies based on IR Evaluation Measures";
in: "ECIR 2017: Advances in Information Retrieval", herausgegeben von: Springer; Springer, Cham, Cham, 2017, ISBN: 978-3-319-56608-5, S. 357 - 368.

Zusätzliche Informationen

Last changed: 
15.12.2017 16:50:35
Accepted: 
Accepted
TU Id: 
264549
Invited: 
Department Focus: 
Business Informatics
Author List: 
A. Lipani, J. Palotti, M. Lupu, F. Piroi, G. Zuccon, A. Hanbury
Abstract German: