Fixed Budget Pooling Strategies based on Fusion Methods

Type: 
Proceedings contribution
Proceedings: 
Proceedings of the Symposium on Applied Computing
Publisher: 
ACM
Pages: 
919 - 924
ISBN: 
ISBN: 978-1-4503-4486-9
Year: 
2017
Abstract: 
The empirical nature of Information Retrieval (IR) mandates strong experimental practices. The Cranfield/TREC evaluation paradigm represents a keystone of such experimental practices. Within this paradigm, the generation of relevance judgments has been the subject of intense scientific investigation. This is because, on one hand, consistent, precise and numerous judgements are key to reduce evaluation uncertainty and test collection bias; on the other hand, however, relevance judgements are costly to collect. The selection of which documents to judge for relevance (known as pooling) has therefore great impact in IR evaluation. In this paper, we contribute a set of 8 novel pooling strategies based on retrieval fusion methods. We show that the choice of the pooling strategy has significant effects on the cost needed to obtain an unbiased test collection; we also identify the best performing pooling strategy according to three evaluation measure.
TU Focus: 
Computational Science and Engineering
Reference: 

A. Lipani, M. Lupu, J. Palotti, G. Zuccon, A. Hanbury:
"Fixed Budget Pooling Strategies based on Fusion Methods";
in: "Proceedings of the Symposium on Applied Computing", ACM, New York, NY, USA, 2017, ISBN: 978-1-4503-4486-9, S. 919 - 924.

Zusätzliche Informationen

Last changed: 
15.12.2017 17:34:07
TU Id: 
264550
Accepted: 
Accepted
Invited: 
Department Focus: 
Author List: 
A. Lipani, M. Lupu, J. Palotti, G. Zuccon, A. Hanbury
Abstract German: