Fast Dawid-Skene

Paper | arXiv | Code | Slides | Supplementary Results

Implementation of the Fast Dawid-Skene and Hybrid algorithms described in the paper:

Vaibhav B Sinha, Sukrut Rao, Vineeth N Balasubramanian. Fast Dawid-Skene: A Fast Vote Aggregation Scheme for Sentiment Classification. In Workshop on Issues of Sentiment Discovery and Opinion Mining (WISDOM) at the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) 2018, August 2018.

Implementations of the Dawid-Skene (Dawid and Skene, 1979) and Majority Voting algorithms are also provided.

These algorithms can be used to aggregate crowd-sourced labels to estimate the true labels. Given the labels of a data point from many annotators out of a set of classes, the algorithms output the most likely correct class for the data point.

Setup

Prerequisites

The prerequisites are: * Python 2.7 or 3.4-3.6 * pip

Setting up dependencies

All other dependencies can be installed using pip, as $ pip install -r requirements.txt

If tests are to be run, use instead, $ pip install -r requirements-dev.txt

Preparing the data

A description of the data format and the procedure to add a new dataset is given here. A toy dataset is also provided, and can be found here.

Running the program

To run the program, use $ python scripts/fast_dawid_skene.py [OPTIONS]

To view a list of available options along with descriptions, use $ python scripts/fast_dawid_skene.py --help

Example Run

To run on the toy dataset, with two annotators per question, using the FDS algorithm to obtain predictions, use $ python scripts/fast_dawid_skene.py --dataset toy --k 2 --mode aggregate --algorithm FDS --print_result To run using all available annotations for every question, using the FDS algorithm to obtain predictions, use $ python scripts/fast_dawid_skene.py --dataset toy --mode aggregate --algorithm FDS --print_result

Running tests

Tests can be run using pytest, as, $ py.test

License

This code is provided under the MIT License.

Some parts of the code in this file are derived from this implementation, and the original license and copyright notice can be found at the top of the file.

Citation

If the Fast Dawid-Skene / Hybrid algorithms are useful for your research, please cite our paper 2.

Acknowledgements

Parts of the code for the implementation of the algorithms use or derive from code in this implementation.

References

1. A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. J. Royal Stat. Soc. Series C 28, 1 (1979), 20–28.
2. Vaibhav B Sinha, Sukrut Rao, Vineeth N Balasubramanian. Fast Dawid-Skene: A Fast Vote Aggregation Scheme for Sentiment Classification. In Workshop on Issues of Sentiment Discovery and Opinion Mining (WISDOM) at the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) 2018, August 2018.