Verifying Extended Entity Relationship Diagrams with Open Tasks

The verification of Extended Entity Relationship (EER) diagrams and other conceptual models that capture the design<br> of information systems is crucial to ensure reliable systems.<br> To scale up verification processes to larger groups of experts,<br> Human Computation techniques were used focusing primarily on closed tasks, which constrain the number and variety of<br> reported defects in favor of easy aggregation of derived judgements. To address this limitation of closed tasks, in this paper,<br> we investigate EER verification (as instance of a broader family of model verification problems) with open tasks to extend<br> the range of collected results. We also address the challenge<br> of aggregating results of open tasks by proposing a followup HC task for defect validation. We evaluate our approach<br> for HC-based EER Verification with open tasks in a set of<br> experiments conducted with junior developers and show that<br> (1) open tasks allow collecting a variety of insights that go<br> beyond a manually built gold standard while still leading to<br> good performance (F1=60%) and (2) HC-based validation<br> can be reliably used for validating the results of open tasks<br> (F1=84% compared to expert validation).