Data

Tools

Indexes

Applications

Experiments

Open Source Science

The Causal Testing Framework

The Causal Testing Framework - Published in JOSS (2025)

https://github.com/citcom-project/causaltestingframework

Science Score: 100.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 12 DOI reference(s) in README and JOSS metadata
  • ✓
    Academic publication links
    Links to: acm.org, joss.theoj.org
  • ✓
    Committers with academic emails
    6 of 11 committers (54.5%) from academic institutions
  • ○
    Institutional organization owner
  • ✓
    JOSS paper metadata
    Published in Journal of Open Source Software

Scientific Fields

Mathematics Computer Science - 88% confidence
Sociology Social Sciences - 87% confidence
Economics Social Sciences - 85% confidence
Last synced: 4 months ago · JSON representation ·

Repository

A causal inference-driven framework for functional black-box testing of complex software.

Basic Info
Statistics
  • Stars: 18
  • Watchers: 3
  • Forks: 5
  • Open Issues: 16
  • Releases: 51
Created about 4 years ago · Last pushed 4 months ago
Metadata Files
Readme Contributing License Citation

README.md

Causal Testing Framework

A Causal Inference-Driven Software Testing Framework

Project Status: Active – The project has reached a stable, usable state and is being actively developed. example workflow codecov Documentation Status Dynamic TOML Badge PyPI - Version GitHub Licens[schematic.tex](images%2Fschematic.tex)e DOI DOI

Causal testing is a causal inference-driven framework for functional black-box testing. This framework utilises graphical causal inference (CI) techniques for the specification and functional testing of software from a black-box perspective. In this framework, we use causal directed acyclic graphs (DAGs) to express the anticipated cause-effect relationships amongst the inputs and outputs of the system-under-test and the supporting mathematical framework to design statistical procedures capable of making causal inferences. Each causal test case focuses on the causal effect of an intervention made to the system-under test. That is, a prescribed change to the input configuration of the system-under-test that is expected to cause a change to some output(s).

Causal Testing Workflow Causal Testing Workflow

Installation

Requirements

  • Python 3.10, 3.11 and 3.12

  • Microsoft Visual C++ 14.0+ (Windows only).

To install the latest stable release of the Causal Testing Framework:

pip install causal-testing-framework

or if you want to install with the development packages/tools:

pip install causal-testing-framework[dev]

Alternatively, you can install directly via source:

shell git clone https://github.com/CITCOM-project/CausalTestingFramework cd CausalTestingFramework then to install a specific release:

shell git fetch --all --tags --prune git checkout tags/<tag> -b <branch> pip install . # For core API only pip install -e . # For editable install, useful for development work For more information on how to use the Causal Testing Framework, please refer to our documentation.

[!NOTE] We recommend you use a 64 bit OS (standard in most modern machines) as we have had reports of the installation crashing on some 32 bit Debian installations.

Usage

[!NOTE] Example usage can be found in the examples directory.

  1. To run the causal testing framework, you need some runtime data from your system, some causal test cases, and a causal DAG that specifies the expected causal relationships between the variables in your runtime data (and any other relevant variables that are not recorded in the data but are known to be relevant).

  2. If you do not already have causal test cases, you can convert your causal DAG to causal tests by running the following command.

python -m causal_testing generate --dag_path $PATH_TO_DAG --output_path $PATH_TO_TESTS

  1. You can now execute your tests by running the following command. python -m causal_testing test --dag_path $PATH_TO_DAG --data_paths $PATH_TO_DATA --test_config $PATH_TO_TESTS --output $OUTPUT The results will be saved for inspection in a JSON file located at $OUTPUT. In the future, we hope to add a visualisation tool to assist with this.

How to Cite

If you use our framework in your work, please cite the following:

This research has used version X.Y.Z (software citation) of the Causal Testing Framework (paper citation).

The paper citation should be the Causal Testing Framework paper, and the software citation should contain the specific Figshare DOI of the version used in your work.

BibTeX Citations
Paper ``` @ARTICLE{Clark_etal_2023, author = {Clark, Andrew G. and Foster, Michael and Prifling, Benedikt and Walkinshaw, Neil and Hierons, Robert M. and Schmidt, Volker and Turner, Robert D.}, title = {Testing Causality in Scientific Modelling Software}, year = {2023}, publisher = {Association for Computing Machinery}, url = {https://doi.org/10.1145/3607184}, doi = {10.1145/3607184}, journal = {ACM Trans. Softw. Eng. Methodol.}, month = {jul}, keywords = {Software Testing, Causal Testing, Causal Inference} } ```
Software (example) ``` @ARTICLE{Wild2023, author = {Foster, Michael and Clark, Andrew G. and Somers, Richard and Wild, Christopher and Allian, Farhad and Hierons, Robert M. and Wagg, David and Walkinshaw, Neil}, title = {CITCOM Software Release}, year = {2023}, month = {nov}, url = {https://orda.shef.ac.uk/articles/software/CITCOM_Software_Release/24427516}, doi = {10.15131/shef.data.24427516.v1} } ```

How to Contribute

To contribute to our work, please ensure the following:

  1. Fork the repository into your own GitHub account, and clone it to your local machine.
  2. Create a new branch in your forked repository. Give this branch an appropriate name, and create commits that describe the changes.
  3. Push your changes to your new branch in your remote fork, compare with CausalTestingFramework/main, and ensure any conflicts are resolved.
  4. Create a draft pull request from your branch, and ensure you have linked it to any relevant issues in your description.

We use the unittest module to develop our tests and the pytest framework as our test discovery, pylint for our code analyser, and black for our code formatting. To find the other (optional) developer dependencies, please check pyproject.toml.

Acknowledgements

The Causal Testing Framework is supported by the UK's Engineering and Physical Sciences Research Council (EPSRC), with the project name CITCOM - "Causal Inference for Testing of Computational Models" under the grant EP/T030526/1.

Owner

  • Name: CITCOM-project
  • Login: CITCOM-project
  • Kind: organization

JOSS Publication

The Causal Testing Framework
Published
March 06, 2025
DOI
Volume 10, Issue 107, Page 7739
Authors
Michael Foster ORCID
University of Sheffield, UK
Andrew Clark ORCID
University of Sheffield, UK
Christopher Wild ORCID
University of Sheffield, UK
Farhad Allian ORCID
University of Sheffield, UK
Robert Turner ORCID
University of Sheffield, UK
Richard Somers ORCID
University of Sheffield, UK
Nicholas Latimer ORCID
University of Sheffield, UK
Neil Walkinshaw ORCID
University of Sheffield, UK
Robert M. Hierons ORCID
University of Sheffield, UK
Editor
Daniel S. Katz ORCID
Tags
causal testing causal inference causality software testing metamorphic testing
View PDF Review Thread Software Archive

Citation (CITATION.cff) 

cff-version: "1.2.0"
authors:
- family-names: Foster
  given-names: Michael
  orcid: "https://orcid.org/0000-0001-8233-9873"
- family-names: Clark
  given-names: Andrew
  orcid: "https://orcid.org/0000-0002-6830-0566"
- family-names: Wild
  given-names: Christopher
  orcid: "https://orcid.org/0009-0009-1195-1497"
- family-names: Allian
  given-names: Farhad
  orcid: "https://orcid.org/0000-0002-4569-0370"
- family-names: Turner
  given-names: Robert
  orcid: "https://orcid.org/0000-0002-1353-1404"
- family-names: Somers
  given-names: Richard
  orcid: "https://orcid.org/0009-0009-1195-1497"
- family-names: Latimer
  given-names: Nicholas
  orcid: "https://orcid.org/0000-0001-5304-5585"
- family-names: Walkinshaw
  given-names: Neil
  orcid: "https://orcid.org/0000-0003-2134-6548"
- family-names: Hierons
  given-names: Robert M.
  orcid: "https://orcid.org/0000-0003-2134-6548"
contact:
- family-names: Foster
  given-names: Michael
  orcid: "https://orcid.org/0000-0001-8233-9873"
doi: 10.15131/shef.data.28539398.v1
message: If you use this software, please cite our article in the
  Journal of Open Source Software.
preferred-citation:
  authors:
  - family-names: Foster
    given-names: Michael
    orcid: "https://orcid.org/0000-0001-8233-9873"
  - family-names: Clark
    given-names: Andrew
    orcid: "https://orcid.org/0000-0002-6830-0566"
  - family-names: Wild
    given-names: Christopher
    orcid: "https://orcid.org/0009-0009-1195-1497"
  - family-names: Allian
    given-names: Farhad
    orcid: "https://orcid.org/0000-0002-4569-0370"
  - family-names: Turner
    given-names: Robert
    orcid: "https://orcid.org/0000-0002-1353-1404"
  - family-names: Somers
    given-names: Richard
    orcid: "https://orcid.org/0009-0009-1195-1497"
  - family-names: Latimer
    given-names: Nicholas
    orcid: "https://orcid.org/0000-0001-5304-5585"
  - family-names: Walkinshaw
    given-names: Neil
    orcid: "https://orcid.org/0000-0003-2134-6548"
  - family-names: Hierons
    given-names: Robert M.
    orcid: "https://orcid.org/0000-0003-2134-6548"
  date-published: 2025-03-06
  doi: 10.21105/joss.07739
  issn: 2475-9066
  issue: 107
  journal: Journal of Open Source Software
  publisher:
    name: Open Journals
  start: 7739
  title: The Causal Testing Framework
  type: article
  url: "https://joss.theoj.org/papers/10.21105/joss.07739"
  volume: 10
title: The Causal Testing Framework

GitHub Events

Total
  • Create event: 38
  • Issues event: 35
  • Release event: 14
  • Watch event: 7
  • Delete event: 17
  • Issue comment event: 116
  • Push event: 203
  • Pull request review event: 75
  • Pull request review comment event: 40
  • Pull request event: 60
  • Fork event: 2
Last Year
  • Create event: 38
  • Issues event: 35
  • Release event: 14
  • Watch event: 7
  • Delete event: 17
  • Issue comment event: 116
  • Push event: 203
  • Pull request review event: 75
  • Pull request review comment event: 40
  • Pull request event: 60
  • Fork event: 2

Committers

Last synced: 5 months ago

All Time
  • Total Commits: 1,382
  • Total Committers: 11
  • Avg Commits per committer: 125.636
  • Development Distribution Score (DDS): 0.653
Past Year
  • Commits: 250
  • Committers: 5
  • Avg Commits per committer: 50.0
  • Development Distribution Score (DDS): 0.264
Top Committers
Name Email Commits
cwild-UoS 9****S 479
Michael Foster m****r@s****k 455
AndrewC19 a****2@s****k 175
f-allian a****d@g****m 123
Richard Somers r****1@s****k 65
Robert (Bob) Turner r****r@s****k 46
cwild-UoS c****d@s****k 33
bobturneruk 3****k 2
Daniel S. Katz d****z@i****g 2
Sam Grayson s****m@s****e 1
James Oyaide 5****r 1
Committer Domains (Top 20 + Academic)
sheffield.ac.uk: 5 samgrayson.me: 1 ieee.org: 1

Issues and Pull Requests

Last synced: 4 months ago

All Time
  • Total issues: 125
  • Total pull requests: 184
  • Average time to close issues: 9 months
  • Average time to close pull requests: 13 days
  • Total issue authors: 9
  • Total pull request authors: 11
  • Average comments per issue: 1.31
  • Average comments per pull request: 2.66
  • Merged pull requests: 157
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 19
  • Pull requests: 67
  • Average time to close issues: 26 days
  • Average time to close pull requests: 9 days
  • Issue authors: 4
  • Pull request authors: 6
  • Average comments per issue: 0.68
  • Average comments per pull request: 2.3
  • Merged pull requests: 49
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • jmafoster1 (39)
  • cwild-UoS (23)
  • f-allian (18)
  • christopher-wild (12)
  • rsomers1998 (11)
  • bobturneruk (9)
  • AndrewC19 (8)
  • neilwalkinshaw (2)
  • akothen (1)
Pull Request Authors
  • jmafoster1 (77)
  • f-allian (51)
  • cwild-UoS (36)
  • christopher-wild (17)
  • AndrewC19 (8)
  • rsomers1998 (8)
  • bobturneruk (7)
  • charmoniumQ (2)
  • Jimazcoder (2)
  • neilwalkinshaw (2)
  • danielskatz (2)
Top Labels
Issue Labels
enhancement (32) json frontend (14) bug (9) documentation (6) Development (4) help wanted (2) good first issue (1) duplicate (1) Testing (1)
Pull Request Labels
enhancement (32) bug (12) documentation (6) Development (2)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 425 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 38
  • Total maintainers: 2
pypi.org: causal-testing-framework

A framework for causal testing using causal directed acyclic graphs.

  • Versions: 38
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 425 Last month
Rankings
Dependent packages count: 6.6%
Downloads: 12.7%
Average: 18.3%
Stargazers count: 18.6%
Forks count: 23.2%
Dependent repos count: 30.6%
Maintainers (2)
cwild jmafoster1
Last synced: 4 months ago

Dependencies

docs/source/requirements.txt pypi
  • sphinx ==4.2.0
  • sphinx-autoapi ==1.8.4
  • sphinx-rtd-theme ==1.0.0
.github/workflows/ci-tests.yaml actions
  • actions/checkout v2 composite
  • codecov/codecov-action v2 composite
  • conda-incubator/setup-miniconda v2 composite
.github/workflows/lint-format.yaml actions
  • actions/checkout v2 composite
  • actions/upload-artifact v2 composite
  • oxsecurity/megalinter/flavors/python v6 composite
.github/workflows/publish-to-pypi.yaml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
  • pypa/gh-action-pypi-publish release/v1 composite
pyproject.toml pypi
  • econml ~=0.12
  • fitter ~=1.4
  • lhsmdu ~=1.1
  • networkx ~=2.6
  • numpy ~=1.23
  • pandas ~=1.3
  • pydot ~=1.4
  • scikit_learn ~=1.1
  • scipy ~=1.7
  • statsmodels ~=0.13
  • tabulate ~=0.8
  • z3_solver ~=4.11.2
.github/workflows/figshare.yaml actions
  • figshare/github-upload-action v1.1 composite