https://github.com/amazon-science/background-summaries
Repository for "Background Summarization of Event Timelines"
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (10.5%) to scientific vocabulary
Repository
Repository for "Background Summarization of Event Timelines"
Basic Info
Statistics
- Stars: 4
- Watchers: 5
- Forks: 1
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Background Summarization of Event Timelines (EMNLP 2023)
This is the repository for the EMNLP 2023 paper "Background Summarization of Event Timelines" by Adithya Pratapa, Kevin Small and Markus Dreyer. The image below provides an overview of the background summarization task.
Dataset
Background summarization dataset is available under data, as well as on Hugging Face datasets.
Training and inference
T5-based systems
We experiment with Flan-T5-XL and Long-T5-TGlobal-XL. For Flan-T5-XL, we explore both generic and query-focused setups. See configs/train.conf for supported model configurations.
```bash
example flan-t5-xl training using deepspeed
bash bash_scripts/t5/train.sh flan-t5-xl 8888 ```
For inference, set the checkpoint path in configs/eval.conf and run the evaluation script.
```bash
example flan-t5-xl inference
bash bash_scripts/t5/eval.sh flan-t5-xl ```
GPT-based systems
We experiment with zero-shot inference with GPT-3.5. See configs/gpt.conf for supported model configurations.
bash
bash bash_scripts/gpt/predict.sh gpt-3.5-turbo
Background Utility Score (BUS)
We propose a new QA-based evaluation metric that measures the utility of a background summary for answering questions about a news update. See the illustration below.
See src/bus/bus.py for details on GPT-3.5 and GPT-4 based BUS metrics.
Human and BUS evaluation data
results contains the data from our Mechanical Turk and BUS evaluations. For the 1,000 news updates from test set, it includes human-written and system-generated backgrounds. It includes results from best-worst ratings, BUS--human, BUS--GPT-3.5 and BUS--GPT-4.
MTurk setup
See src/mturk for details on MTurk setup.
Model checkpoints and predictions
To download the model checkpoints and predictions,
bash
URL=https://d1f9rvlwrb54wt.cloudfront.net/background-summaries
wget $URL/models-flan-t5.tgz # flan-t5-xl (file size: ~10G)
wget $URL/models-flan-t5-ift.tgz # flan-t5-xl-ift, flan-t5-xl-ift-ents (file size: ~20G)
wget $URL/models-gpt-anns.tgz # gpt-3.5-turbo, gpt-3.5-turbo-cond-ents, human annotators (file size: ~5M)
wget $URL/models-long-t5.tgz # long-t5-tglobal-xl (file size: ~10G)
Security
See CONTRIBUTING for more information.
License
This project is licensed under the CC-BY-NC-4.0 License. See the LICENSE file.
Reference
You can cite our paper as follows:
@inproceedings{pratapa-etal-2023-background,
title = "Background Summarization of Event Timelines",
author = "Pratapa, Adithya and Small, Kevin and Dreyer, Markus",
booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
publisher = "Association for Computational Linguistics",
year="2023"
}
Owner
- Name: Amazon Science
- Login: amazon-science
- Kind: organization
- Website: https://amazon.science
- Twitter: AmazonScience
- Repositories: 80
- Profile: https://github.com/amazon-science
GitHub Events
Total
- Watch event: 2
Last Year
- Watch event: 2
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 0
- Total pull requests: 2
- Average time to close issues: N/A
- Average time to close pull requests: about 4 hours
- Total issue authors: 0
- Total pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.0
- Merged pull requests: 2
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
Pull Request Authors
- adithya7 (2)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- Unidecode *
- absl-py *
- accelerate *
- bert_score *
- bs4 *
- datasets *
- deepspeed *
- evaluate *
- pyhocon *
- pyrouge *
- pytest *
- readability-lxml *
- rouge_score *
- seaborn *
- sentencepiece *
- spacy *
- transformers *
- ujson *
- wandb *