https://github.com/awslabs/fever
FEVER (Fact Extraction and VERification) Annotation Platform and Baselines
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (8.8%) to scientific vocabulary
Repository
FEVER (Fact Extraction and VERification) Annotation Platform and Baselines
Basic Info
- Host: GitHub
- Owner: awslabs
- License: apache-2.0
- Language: HTML
- Default Branch: master
- Size: 729 KB
Statistics
- Stars: 109
- Watchers: 8
- Forks: 16
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Fact Extraction and VERification
This is the main repository for the dataset and experiments presented in our NAACL2018 paper: FEVER: A large-scale dataset for Fact Extraction and VERification.
Unlike other tasks and despite recent interest, research in textual claim verification has been hindered by the lack of large-scale manually annotated datasets. In this paper we introduce a new publicly available dataset for verification against textual sources, FEVER: Fact Extraction and VERification. It consists of 185,441 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from. The claims are classified as Supported, Refuted or NotEnoughInfo by annotators achieving 0.6841 in Fleiss κ. For the first two classes, the annotators also recorded the sentence(s) forming the necessary evidence for their judgment. To characterize the challenge of the dataset presented, we develop a pipeline approach using both baseline and state-of-the-art components and compare it to suitably designed oracles. The best accuracy we achieve on labeling a claim accompanied by the correct evidence is 31.87%, while if we ignore the evidence we achieve 50.91%. Thus we believe that FEVER is a challenging testbed that will help stimulate progress on claim verification against textual sources
This repository contains the baseline experiments, the scorer as submodules as well as the code used to generate and prepare the dataset.
Get Started
git clone https://github.com/awslabs/fever.git
cd fever
git submodule init
git submodule update
Download the Data
Visit http://fever.ai to download the data and find out more about the shared task.
Run the Code
Owner
- Name: Amazon Web Services - Labs
- Login: awslabs
- Kind: organization
- Location: Seattle, WA
- Website: http://amazon.com/aws/
- Repositories: 914
- Profile: https://github.com/awslabs
AWS Labs
GitHub Events
Total
- Watch event: 17
- Fork event: 4
Last Year
- Watch event: 17
- Fork event: 4
Issues and Pull Requests
Last synced: about 2 years ago
All Time
- Total issues: 1
- Total pull requests: 10
- Average time to close issues: 2 days
- Average time to close pull requests: about 2 hours
- Total issue authors: 1
- Total pull request authors: 4
- Average comments per issue: 3.0
- Average comments per pull request: 0.1
- Merged pull requests: 9
- Bot issues: 0
- Bot pull requests: 2
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- phoenixsecularbird (1)
Pull Request Authors
- j6mes (6)
- dependabot[bot] (4)
- bopopescu (1)
- christos-c (1)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- Flask ==1.0
- Flask-cors *
- PyMySQL ==0.7.9
- SQLAlchemy ==1.3.0
- boto3 *
- botocore >=1.9.23
- jnius ==1.1.0
- mwparserfromhell ==0.4.4
- nltk *
- statsmodels *
- tqdm ==4.14.0