countergen

A framework for generating counterfactual datasets, evaluating NLP models, and editing models to reduce bias

https://github.com/fabienroger/countergen

Repository

A framework for generating counterfactual datasets, evaluating NLP models, and editing models to reduce bias

README.rst

CounterGen
==========

**CounterGen** is a framework for generating counterfactual datasets, evaluating NLP models, and editing models to reduce bias.
It provides powerful defaults, while offering simple ways to use your own data, data augmentation techniques, models, and evaluation metrics.

* ``countergen`` is a lightweight Python module which helps you generate counterfactual datasets and evaluate bias of models available locally or through an API. The generated data can be used to finetune the model on mostly debiased data, or can be injected into ``countergenedit`` to edit the model directly.
* ``countergenedit`` is a Python module which adds methods to easily evaluate PyTorch text generation models as well as text classifiers. It provides tools to analyze model activation and edit model to reduce bias.

Read the docs here: https://fabienroger.github.io/Countergen/

How ``countergen`` helps you evaluate model bias
----------------------------------------------

.. image:: docs/countergen_explanation.png
  :width: 700
  :align: center
  :alt: Evaluation explanation

|

How ``countergenedit`` helps you edit models
----------------------------------------------

.. image:: docs/countergenedit_explanation.png
  :width: 700
  :align: center
  :alt: Edition explanation

|

Minimal examples
---------------------

Model evaluation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python
  
  import countergen
  augmented_ds = countergen.AugmentedDataset.from_default("male-stereotypes")
  api_model = countergen.api_to_generative_model("davinci") # Evaluate GPT-3
  model_evaluator = countergen.get_generative_model_evaluator(api_model)
  countergen.evaluate_and_print(augmented_ds.samples, model_evaluator)


*(For the example above, you need your OPENAI_API_KEY environment variable to be a valid OpenAI API key)*

Data augmentation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python
  
  import countergen
  ds = countergen.Dataset.from_jsonl("my_data.jsonl")
  augmenters = [countergen.SimpleAugmenter.from_default("gender")]
  augmented_ds = ds.augment(augmenters)
  augmented_ds.save_to_jsonl("my_data_augmented.jsonl")


Model editing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python
  
  import countergen as cg
  import countergenedit as cge
  from transformers import GPT2LMHeadModel
  augmented_ds = cg.AugmentedDataset.from_default("male-stereotypes")
  model = GPT2LMHeadModel.from_pretrained("gpt2")
  layers = cge.get_mlp_modules(model, [2, 3])
  activation_ds = cge.ActivationsDataset.from_augmented_samples(
    augmented_ds.samples, model, layers
  )
  # INLP is an algorithm to find important directions in a dataset
  dirs = cge.inlp(activation_ds)
  configs = cge.get_edit_configs(layers, dirs)
  new_model = cge.edit_model(model, configs=configs)


Work We Use
------------------------------------

- LLMD `(Fryer, 2022) `_, to augment data using large language models;
- INLP `(Ravfogel, 2020) `_ and RLACE `(Ravfogel, 2022) `_, to find key directions in neural activations;
- Stereoset `(Nadeem, 2020) `_, a large collection of stereotypes;
- The "Double bind experiment" `(Heilman, 2007) `_, an experiment about bias in humans which can also be conducted with large language models, and `(May, 2019) `_, which provides the exact data we use;
- `OpenAI's API `_, to run inferences on large languages models;
- `De Gibert, 2018 `_, which provides data about hate speech;
- `nltk `_, and Jörg Michael's `gender.c `_, which contain datasets about the gender and origin of first names;
- BigBench's `Social Bias from Sentence Probability `_, which provides evaluation data and metrics.

Owner

Citation (citations.bib)

gpt3
@article{brown2020language,
  title={Language models are few-shot learners},
  author={Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and others},
  journal={Advances in neural information processing systems},
  volume={33},
  pages={1877--1901},
  year={2020}
}
doublebind
@article{heilman2007women,
  title={Why are women penalized for success at male tasks?: the implied communality deficit.},
  author={Heilman, Madeline E and Okimoto, Tyler G},
  journal={Journal of applied psychology},
  volume={92},
  number={1},
  pages={81},
  year={2007},
  publisher={American Psychological Association}
}
stereotypes
@article{nadeem2020stereoset,
  title={Stereoset: Measuring stereotypical bias in pretrained language models},
  author={Nadeem, Moin and Bethke, Anna and Reddy, Siva},
  journal={arXiv preprint arXiv:2004.09456},
  year={2020}
}
llmd
@article{fryer2022flexible,
  title={Flexible text generation for counterfactual fairness probing},
  author={Fryer, Zee and Axelrod, Vera and Packer, Ben and Beutel, Alex and Chen, Jilin and Webster, Kellie},
  journal={arXiv preprint arXiv:2206.13757},
  year={2022}
}
rlace
@inproceedings{ravfogel2022linear,
  title={Linear adversarial concept erasure},
  author={Ravfogel, Shauli and Twiton, Michael and Goldberg, Yoav and Cotterell, Ryan D},
  booktitle={International Conference on Machine Learning},
  pages={18400--18421},
  year={2022},
  organization={PMLR}
}
inlp
@article{ravfogel2020null,
  title={Null it out: Guarding protected attributes by iterative nullspace projection},
  author={Ravfogel, Shauli and Elazar, Yanai and Gonen, Hila and Twiton, Michael and Goldberg, Yoav},
  journal={arXiv preprint arXiv:2004.07667},
  year={2020}
}
for the gender occupations
@incollection{lu2020gender,
  title={Gender bias in neural natural language processing},
  author={Lu, Kaiji and Mardziel, Piotr and Wu, Fangjing and Amancharla, Preetam and Datta, Anupam},
  booktitle={Logic, Language, and Security},
  pages={189--202},
  year={2020},
  publisher={Springer}
}
for the names by country
@incollection{michaelgender,
  title={gender.c},
  author={Michael, Jörg},
  url={https://www.autohotkey.com/board/topic/20260-gender-verification-by-forename-cmd-line-tool-db/},
}
nltk, for the names
@book{bird2009natural,
  title={Natural language processing with Python: analyzing text with the natural language toolkit},
  author={Bird, Steven and Klein, Ewan and Loper, Edward},
  year={2009},
  publisher={" O'Reilly Media, Inc."}
}
hatespeech
@article{de2018hate,
  title={Hate speech dataset from a white supremacy forum},
  author={De Gibert, Ona and Perez, Naiara and Garc{\'\i}a-Pablos, Aitor and Cuadros, Montse},
  journal={arXiv preprint arXiv:1809.04444},
  year={2018}
}
doublebind data
@article{may2019measuring,
  title={On measuring social biases in sentence encoders},
  author={May, Chandler and Wang, Alex and Bordia, Shikha and Bowman, Samuel R and Rudinger, Rachel},
  journal={arXiv preprint arXiv:1903.10561},
  year={2019}
}
Social Bias from Sentence Probability data & metric
@inproceedings{ghazal2013bigbench,
  title={Bigbench: Towards an industry standard benchmark for big data analytics},
  author={Ghazal, Ahmad and Rabl, Tilmann and Hu, Minqing and Raab, Francois and Poess, Meikel and Crolotte, Alain and Jacobsen, Hans-Arno},
  booktitle={Proceedings of the 2013 ACM SIGMOD international conference on Management of data},
  pages={1197--1208},
  year={2013}
}

GitHub Events

Committers

Last synced: over 3 years ago

Issues and Pull Requests

Last synced: 11 months ago

Packages