YASO: A Targeted Sentiment Analysis Evaluation Dataset for Open-Domain Reviews

Scope

This repository contains:

(1) The YASO evaluation dataset for targeted sentiment analysis (TSA).

(2) Code for evaluating the output of TSA systems on YASO.

(3) New! The TSA-MD dataset, described here.

(4) New! Domain labels added to the YASO evaluation datast, see the changelog for details.

Usage

Data: Some of the sentences annotated in YASO are taken from other datasets that cannot be re-distributed in clear text. To obtain the original texts please follow the instructions here.

Evaluation code:

Using pip: pip install git+ssh://git@github.com/IBM/yaso-tsa.git#egg=yaso-tsa

Alternatively, you can first clone the code, and install the requirements:

commandline 1. git clone git@github.com:IBM/yaso-tsa.git 2. cd yaso-tsa/yaso_tsa 3. pip install -r requirements.txt

Use the module yaso_tsa.evaluate_tsa.

For example, run the following command from the main directory of the repository:

commandline python -m yaso_tsa.evaluate_tsa --predictions_path tests/data/test_data.json --labels_path tests/data/test_labels.json

The expected output should be similar to:

text [MainThread] 2021-09-13:16:37:15,137 INFO [evaluate_tsa.py:34] Loaded labeled data: <TsaLabels labeled: 4, sentences: 3> [MainThread] 2021-09-13:16:37:15,190 INFO [evaluate_tsa.py:44] precision=0.6666666666666666 [MainThread] 2021-09-13:16:37:15,190 INFO [evaluate_tsa.py:44] recall=0.6666666666666666 [MainThread] 2021-09-13:16:37:15,190 INFO [evaluate_tsa.py:44] F1=0.6666666666666666

Citing YASO

If you are using YASO in a publication, please cite the following paper:

Matan Orbach, Orith Toledo-Ronen, Artem Spector, Ranit Aharonov, Yoav Katz and Noam Slonim. 2021. YASO: A Targeted Sentiment Analysis Evaluation Dataset for Open-Domain Reviews. EMNLP.

Contributing

This project welcomes external contributions, if you would like to contribute please see further instructions here

Pull requests are very welcome! Make sure your patches are well tested. Ideally create a topic branch for every separate change you make. For example:

1. Fork the repo
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Added some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Changelog

Major changes are documented here.

Notes

If you have any questions or issues you can create a new issue here.

License

This code is distributed under Apache License 2.0. If you would like to see the detailed LICENSE click here.

Authors

The YASO dataset was collected by Matan Orbach, Orith Toledo-Ronen, Artem Spector, Ranit Aharonov, Yoav Katz and Noam Slonim.

The evaluation code was written by Matan Orbach and Artem Spector.