Fixed Budget Pooling Strategies based on Fusion Methods

Authors: 
Aldo Lipani
Mihai Lupu
João Palotti
Guido Zuccon
Type: 
Proceedings contribution
Proceedings: 
Proceedings of the Symposium on Applied Computing
Publisher: 
ACM
Pages: 
919 - 924
Year: 
2017
ISBN: 
ISBN: 978-1-4503-4486-9
Abstract: 
The empirical nature of Information Retrieval (IR) mandates strong experimental practices. The Cranfield/TREC evaluation paradigm represents a keystone of such experimental practices. Within this paradigm, the generation of relevance judgments has been the subject of intense scientific investigation. This is because, on one hand, consistent, precise and numerous judgements are key to reduce evaluation uncertainty and test collection bias; on the other hand, however, relevance judgements are costly to collect. The selection of which documents to judge for relevance (known as pooling) has therefore great impact in IR evaluation. In this paper, we contribute a set of 8 novel pooling strategies based on retrieval fusion methods. We show that the choice of the pooling strategy has significant effects on the cost needed to obtain an unbiased test collection; we also identify the best performing pooling strategy according to three evaluation measure.
TU Focus: 
Information and Communication Technology
Reference: 

A. Lipani, M. Lupu, J. Palotti, G. Zuccon, A. Hanbury:
"Fixed Budget Pooling Strategies based on Fusion Methods";
in: "Proceedings of the Symposium on Applied Computing", ACM, New York, NY, USA, 2017, ISBN: 978-1-4503-4486-9, S. 919 - 924.

Zusätzliche Informationen

Last changed: 
15.12.2017 17:34:07
Accepted: 
Accepted
TU Id: 
264550
Invited: 
Department Focus: 
Business Informatics
Author List: 
A. Lipani, M. Lupu, J. Palotti, G. Zuccon, A. Hanbury
Abstract German: