Metasyn
Metasyn: Transparent Generation of Synthetic Tabular Data with Privacy Guarantees - Published in JOSS (2025)
Science Score: 98.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
✓DOI references
Found 6 DOI reference(s) in README and JOSS metadata -
✓Academic publication links
Links to: joss.theoj.org, zenodo.org -
○Committers with academic emails
-
○Institutional organization owner
-
✓JOSS paper metadata
Published in Journal of Open Source Software
Keywords
Scientific Fields
Repository
Transparent and privacy-friendly synthetic data generation
Basic Info
- Host: GitHub
- Owner: sodascience
- License: mit
- Language: Python
- Default Branch: develop
- Homepage: https://metasyn.readthedocs.io
- Size: 7.73 MB
Statistics
- Stars: 43
- Watchers: 3
- Forks: 9
- Open Issues: 9
- Releases: 18
Topics
Metadata Files
README.md
Transparent and privacy-friendly synthetic data generation
Generate synthetic tabular data in a transparent, understandable, and privacy-friendly way. Metasyn makes it possible for owners of sensitive data to create test data, do open science, improve code reproducibility, encourage data reuse, and enhance accessibility of their datasets, without worrying about leaking private information.
With metasyn you can fit a model to an existing dataframe, save it to a transparent and auditable .json file, and synthesize a dataframe that looks a lot like the real one. In contrast to most other synthetic data software, we make the explicit choice to strictly limit the statistical information in our model in order to adhere to the highest privacy standards.
Highlights
- 👋 Accessible. Metasyn is designed to be easy to use and understand, and we do our best to be welcoming to newcomers and novice users. Let us know if we can improve!
- ✨ Fully featured. Out of the box, metasyn natively handles a wide range of data types, missing values, categorical data, key columns with unique values, and structured strings such as postcodes or identifiers.
- 🔎 Transparent. With metasyn you share not only synthetic data, but also the model and settings used to create it through a traceable, auditable metadata format. Everyone can read and understand what the model does; it is crystal clear which information becomes public.
- 🔐 Private. By default, metasyn does not incorporate multivariate information, meaning less risk of privacy issues such as identity, attribute, or group disclosure. On top of this, we support privacy plugins such as our own disclosure control plugin to further enhance privacy in critically sensitive situations.
- 🔗 Integrated. We integrate closely with popular, modern tools in the python ecosystem, building on the wonderful polars dataframe library (pandas is supported too), as well as faker to generate localized data for names, emails, and phone numbers, and more.
- 📦 Extensible. Are you missing features? Do you have a different definition of privacy? Our plugin system allows you (or your organisation) to create their own extension to adjust metasyn to what you need. Or you can contribute directly to the project.
Installation
Metasyn can be installed directly from PyPI using the following command in the terminal:
sh
pip install metasyn
The latest (possibly unstable) development version can be installed directly from GitHub like so:
sh
pip install git+https://github.com/sodascience/metasyn
Usage
To generate synthetic data, metasyn first needs to fit a MetaFrame to the data which can then be used to produce new synthetic rows:
The above image closely matches the Python code:
```python import polars as pl from metasyn import MetaFrame, demo_file
Get the csv file path for built-in demo dataset
csvpath = demofile("fruit")
Create a polars dataframe from the csv file.
It is important to ensure the data types are correct
when creating your dataframe, especially categorical data!
df = pl.readcsv(csvpath, schema_overrides={ "fruits": pl.Categorical, "cars": pl.Categorical })
Create a MetaFrame from the DataFrame.
mf = MetaFrame.fit_dataframe(df)
Generate a new DataFrame with 5 rows from the MetaFrame.
df_synth = mf.synthesize(5)
This DataFrame can be exported to csv, parquet, excel and more.
dfsynth.writecsv("output.csv") ```
To explore more options and try this out online, take a look at our interactive tutorial:
For more information on how to create dataframes with polars, refer to the Polars documentation. Metasyn also works with pandas dataframes!
Where to go next
- To explore more options and try this out online, take a look at our interactive tutorial:
- As a next step to learn more about generating synthetic data with metasyn we recommend to check out the user guide and other documentation.
- For even more privacy, have a look at our disclosure control plugin.
- Want to create programs that build on metasyn? Take a look at our versioned Docker containers and our CLI.
Contributing
Metasyn is an open-source project, and we welcome contributions from the community, from bug reports & feature requests to code contributions. Read our contributing guidelines for more information and to get started!
Contact
Metasyn is a project by the ODISSEI Social Data Science (SoDa) team. Do you have questions, suggestions, or remarks on the technical implementation? Create an issue in the issue tracker or feel free to contact Erik-Jan van Kesteren or Raoul Schram.
Owner
- Name: SoDa
- Login: sodascience
- Kind: organization
- Location: Utrecht
- Website: http://odissei-data.nl/soda
- Repositories: 17
- Profile: https://github.com/sodascience
ODISSEI Social Data Science Team
JOSS Publication
Metasyn: Transparent Generation of Synthetic Tabular Data with Privacy Guarantees
Authors
Utrecht University, The Netherlands
Utrecht University, The Netherlands
Utrecht University, The Netherlands, ODISSEI: Open Data Infrastructure for Social Science and Economic Innovations, The Netherlands
Editor
Chris Vernon
Tags
synthetic data privacy generative models data managementCitation (CITATION.cff)
cff-version: 1.2.0
title: metasyn
message: >-
If you use this software, please cite it using the
metadata from this file.
type: software
authors:
- given-names: Raoul
family-names: Schram
email: r.d.schram@uu.nl
affiliation: Utrecht University
orcid: 'https://orcid.org/0000-0001-6616-230X'
- given-names: Samuel
family-names: Spithorst
email: s.f.spithorst@uu.nl
orcid: 'https://orcid.org/0009-0000-4140-0658'
affiliation: Utrecht University
- given-names: Erik-Jan
name-particle: van
family-names: Kesteren
email: e.vankesteren1@uu.nl
affiliation: Utrecht University
orcid: 'https://orcid.org/0000-0003-1548-1663'
identifiers:
- type: doi
value: 10.5281/zenodo.7696031
description: Latest archived version of metasyn
repository-code: 'https://github.com/sodascience/metasyn'
url: 'https://metasyn.readthedocs.io/'
abstract: >-
A Python package for generating synthetic data from
tabular datasets, with a focus on privacy and
transparency.
keywords:
- synthetic data
- metadata
- generative model
- data science
- data management
- machine learning
- statistics
license: MIT
GitHub Events
Total
- Create event: 25
- Issues event: 32
- Watch event: 9
- Delete event: 20
- Issue comment event: 26
- Push event: 122
- Gollum event: 2
- Pull request review comment event: 28
- Pull request review event: 37
- Pull request event: 50
Last Year
- Create event: 25
- Issues event: 32
- Watch event: 9
- Delete event: 20
- Issue comment event: 26
- Push event: 123
- Gollum event: 2
- Pull request review comment event: 28
- Pull request review event: 37
- Pull request event: 50
Committers
Last synced: 5 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| qubixes | 4****s | 137 |
| Erik-Jan van Kesteren | e****n@p****e | 58 |
| Samuwhale | s****t@g****m | 55 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 4 months ago
All Time
- Total issues: 122
- Total pull requests: 204
- Average time to close issues: about 2 months
- Average time to close pull requests: 8 days
- Total issue authors: 8
- Total pull request authors: 3
- Average comments per issue: 0.88
- Average comments per pull request: 0.75
- Merged pull requests: 179
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 28
- Pull requests: 66
- Average time to close issues: 29 days
- Average time to close pull requests: 12 days
- Issue authors: 5
- Pull request authors: 2
- Average comments per issue: 0.43
- Average comments per pull request: 0.55
- Merged pull requests: 53
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- vankesteren (59)
- qubixes (44)
- maartenschermer (7)
- chStaiger (4)
- PetrKorab (3)
- Samuwhale (3)
- LindaNab (1)
- misken (1)
Pull Request Authors
- qubixes (139)
- vankesteren (51)
- Samuwhale (14)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 2
-
Total downloads:
- pypi 97 last-month
-
Total dependent packages: 1
(may contain duplicates) -
Total dependent repositories: 1
(may contain duplicates) - Total versions: 19
- Total maintainers: 1
pypi.org: metasynth
[Inactive] Package for creating synthetic datasets while preserving privacy.
- Documentation: https://metasynth.readthedocs.io/
- License: MIT License Copyright (c) 2023 SoDa Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
Latest release: 0.5.999
published over 2 years ago
Rankings
Maintainers (1)
pypi.org: metasyn
Package for creating synthetic datasets while preserving privacy.
- Documentation: https://metasyn.readthedocs.io/
- License: MIT License Copyright (c) 2024 SoDa Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
Latest release: 1.1.0
published about 1 year ago
Rankings
Maintainers (1)
Dependencies
- actions/checkout v4 composite
- actions/setup-python v4 composite
- actions/checkout v4 composite
- actions/setup-python v4 composite
- pypa/gh-action-pypi-publish 7eb3b701d11256e583f5b49899c5e7203deab573 composite
- python 3.11-slim build
- sphinx <7.0.0
- sphinx-autodoc-typehints *
- sphinx-rtd-theme *
- sphinx_copybutton *
- sphinx_inline_tabs *
- sphinxcontrib-napoleon *
- faker *
- importlib-metadata python_version<'3.10'
- importlib-resources python_version<'3.9'
- jsonschema *
- lingua-language-detector *
- numpy >=1.20
- pandas *
- polars >=0.14.17
- pyarrow *
- regexmodel >=0.2.1
- scikit-learn *
- scipy *
- tqdm *
- wget *