FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations (NAACL 2022)

This repository contains the code for the paper "FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations".

FactGraph is an adapter-based method for assessing factuality that decomposes the document and the summary into structured meaning representations (MR):

In FactGraph, summary and document graphs are encoded by a graph encoder with structure-aware adapters, along with text representations using an adapter-based text encoder. Text and graph encoders use the same pretrained model and only the adapters are trained:

Environment

The easiest way to proceed is to create a conda environment: conda create -n factgraph python=3.7 conda activate factgraph

Further, install PyTorch and PyTorch Geometric:

pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html pip install torch-scatter==2.0.9 -f https://data.pyg.org/whl/torch-1.9.0+cu111.html pip install torch-sparse==0.6.12 -f https://data.pyg.org/whl/torch-1.9.0+cu111.html pip install torch-geometric==2.0.3

Install the packages required:

pip install -r requirements.txt

Finally, create the environment for AMR preprocessing:

cd data/preprocess ./create_envs_preprocess.sh cd ../../

FactCollect Dataset

FactCollect is created consolidating the following datasets:

| Dataset | Datapoints | | | ------------- |:-------------:|:-------------:| | Wang et al. (2020) | 953 | Link | Kryscinski et al. (2020) | 1434 | Link | Maynez et al. (2020) | 2500 | Link | Pagnoni et al. (2021) | 4942 | Link

• FactCollect uses two datasets released under licenses.
  • FactCC is under BSD-3. Copyright (c) 2019, Salesforce.com, Inc. All rights reserved.
  • XSum Hallucinations is under CC BY 4.0.

For generating FactCollect dataset, execute:

conda activate factgraph cd data ./create_dataset.sh cd ..

Running trained FactGraph Models

First, download FactGraph trained checkpoints: cd src ./download_trained_models.sh

To run FactGraph: ./evaluate.sh factgraph <file> <gpu_id>

To run FactGraph edge-level: ./evaluate.sh factgraph-edge <file> <gpu_id>

<file> is a JSON line file with the following format: {'summary': summary1, 'article': article1} {'summary': summary2, 'article': article2} ... where 'summary' is a single sentence summary.

Training FactGraph

Preprocess

Convert the dataset into the format required for the model:

cd data/preprocess ./process_dataset_for_model.sh <gpu_id> cd ../../

This step generated AMR graphs using the SPRING model. Check their repository for more details.

Download the pretrained parameters of the adapters: cd src ./download_pretrained_adapters.sh

Training

For training FactGraph using the FactCollect dataset, execute: conda activate factgraph ./train.sh <gpu_id>

Predicting

For predicting, run: ./predict.sh <checkpoint_folder> <gpu_id>

Training FactGraph - Edge-level

Preprocess

Download the files train.tsv and test.tsv from this link provided by Goyal and Durrett (2021). Copy those files to data\edge_level_data

Convert the dataset into the format required for the model:

cd data/preprocess ./process_dataset_for_edge_model.sh <gpu_id> cd ../../

Training

For training FactGraph using the FactCollect dataset, execute: conda activate factgraph ./train_edgelevel.sh <gpu_id>

Predicting

For predicting, run: ./predict_edgelevel.sh <checkpoint_folder> <gpu_id>

Trained Models

A FactGraph checkpoint trained on FactCollect dataset can be found here. Test set results: {'accuracy': 0.89, 'bacc': 0.8904, 'f1': 0.89, 'size': 600, 'cnndm': {'bacc': 0.7717, 'f1': 0.8649, 'size': 370}, 'xsum': {'bacc': 0.6833, 'f1': 0.9304, 'size': 230}}

A FactGraph-edge checkpoint trained on the Maynez dataset can be found here. This checkpoint was selected using the test set. Test set results: {'accuracy': 0.8371, 'bacc': 0.8447, 'f1': 0.8371, 'f1_macro': 0.7362, 'accuracy_edge': 0.6948, 'bacc_edge': 0.6592, 'f1_edge': 0.6948}

Security

See CONTRIBUTING for more information.

License Summary

The documentation is made available under under the CC-BY-NC-4.0 License. See the LICENSE file.

Citation

@inproceedings{ribeiro-etal-2022-factgraph, title = "FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations", author = "Ribeiro, Leonardo F. R. and Liu, Mengwen and Gurevych, Iryna and Dreyer Markus and Bansal, Mohit", booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies", year={2022} }