https://github.com/sdv-dev/sdv
Synthetic data generation for tabular data
Science Score: 59.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
✓DOI references
Found 1 DOI reference(s) in README -
✓Academic publication links
Links to: ieee.org -
✓Committers with academic emails
23 of 51 committers (45.1%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (16.0%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
Synthetic data generation for tabular data
Basic Info
- Host: GitHub
- Owner: sdv-dev
- License: other
- Language: Python
- Default Branch: main
- Homepage: https://docs.sdv.dev/sdv
- Size: 31.8 MB
Statistics
- Stars: 3,150
- Watchers: 43
- Forks: 380
- Open Issues: 154
- Releases: 81
Topics
Metadata Files
README.md
This repository is part of The Synthetic Data Vault Project, a project from DataCebo.
[](https://pypi.org/search/?c=Development+Status+%3A%3A+5+-+Production%2FStable) [](https://pypi.python.org/pypi/SDV) [](https://github.com/sdv-dev/SDV/actions/workflows/unit.yml?query=branch%3Amain) [](https://github.com/sdv-dev/SDV/actions/workflows/integration.yml?query=branch%3Amain) [](https://codecov.io/gh/sdv-dev/SDV) [](https://pepy.tech/project/sdv) [](https://docs.sdv.dev/sdv/demos) [](https://bit.ly/sdv-slack-invite)
Overview
The Synthetic Data Vault (SDV) is a Python library designed to be your one-stop shop for creating tabular synthetic data. The SDV uses a variety of machine learning algorithms to learn patterns from your real data and emulate them in synthetic data.
Features
:brain: Create synthetic data using machine learning. The SDV offers multiple models, ranging from classical statistical methods (GaussianCopula) to deep learning methods (CTGAN). Generate data for single tables, multiple connected tables or sequential tables.
:bar_chart: Evaluate and visualize data. Compare the synthetic data to the real data against a variety of measures. Diagnose problems and generate a quality report to get more insights.
:arrows_counterclockwise: Preprocess, anonymize and define constraints. Control data processing to improve the quality of synthetic data, choose from different types of anonymization and define business rules in the form of logical constraints.
| Important Links | |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------|
| 
Tutorials | Get some hands-on experience with the SDV. Launch the tutorial notebooks and run the code yourself. |
| :book: Docs | Learn how to use the SDV library with user guides and API references. |
| :orange_book: Blog | Get more insights about using the SDV, deploying models and our synthetic data community. |
| 
Community | Join our Slack workspace for announcements and discussions. |
| :computer: Website | Check out the SDV website for more information about the project. |
Install
The SDV is publicly available under the Business Source License. Install SDV using pip or conda. We recommend using a virtual environment to avoid conflicts with other software on your device.
bash
pip install sdv
bash
conda install -c pytorch -c conda-forge sdv
Getting Started
Load a demo dataset to get started. This dataset is a single table describing guests staying at a fictional hotel.
```python from sdv.datasets.demo import download_demo
realdata, metadata = downloaddemo( modality='singletable', datasetname='fakehotelguests') ```
The demo also includes metadata, a description of the dataset, including the data types in each
column and the primary key (guest_email).
Synthesizing Data
Next, we can create an SDV synthesizer, an object that you can use to create synthetic data. It learns patterns from the real data and replicates them to generate synthetic data. Let's use the GaussianCopulaSynthesizer.
```python from sdv.single_table import GaussianCopulaSynthesizer
synthesizer = GaussianCopulaSynthesizer(metadata) synthesizer.fit(data=real_data) ```
And now the synthesizer is ready to create synthetic data!
python
synthetic_data = synthesizer.sample(num_rows=500)
The synthetic data will have the following properties: - Sensitive columns are fully anonymized. The email, billing address and credit card number columns contain new data so you don't expose the real values. - Other columns follow statistical patterns. For example, the proportion of room types, the distribution of check in dates and the correlations between room rate and room type are preserved. - Keys and other relationships are intact. The primary key (guest email) is unique for each row. If you have multiple tables, the connection between a primary and foreign keys makes sense.
Evaluating Synthetic Data
The SDV library allows you to evaluate the synthetic data by comparing it to the real data. Get started by generating a quality report.
```python from sdv.evaluation.singletable import evaluatequality
qualityreport = evaluatequality( realdata, syntheticdata, metadata) ```
``` Generating report ...
(1/2) Evaluating Column Shapes: |████████████████| 9/9 [00:00<00:00, 1133.09it/s]| Column Shapes Score: 89.11%
(2/2) Evaluating Column Pair Trends: |██████████████████████████████████████████| 36/36 [00:00<00:00, 502.88it/s]| Column Pair Trends Score: 88.3%
Overall Score (Average): 88.7% ```
This object computes an overall quality score on a scale of 0 to 100% (100 being the best) as well as detailed breakdowns. For more insights, you can also visualize the synthetic vs. real data.
```python from sdv.evaluation.singletable import getcolumn_plot
fig = getcolumnplot( realdata=realdata, syntheticdata=syntheticdata, columnname='amenitiesfee', metadata=metadata )
fig.show() ```
What's Next?
Using the SDV library, you can synthesize single table, multi table and sequential data. You can also customize the full synthetic data workflow, including preprocessing, anonymization and adding constraints.
To learn more, visit the SDV Demo page.
Credits
Thank you to our team of contributors who have built and maintained the SDV ecosystem over the years!
Citation
If you use SDV for your research, please cite the following paper:
Neha Patki, Roy Wedge, Kalyan Veeramachaneni. The Synthetic Data Vault. IEEE DSAA 2016.
@inproceedings{
SDV,
title={The Synthetic data vault},
author={Patki, Neha and Wedge, Roy and Veeramachaneni, Kalyan},
booktitle={IEEE International Conference on Data Science and Advanced Analytics (DSAA)},
year={2016},
pages={399-410},
doi={10.1109/DSAA.2016.49},
month={Oct}
}
The Synthetic Data Vault Project was first created at MIT's Data to AI Lab in 2016. After 4 years of research and traction with enterprise, we created DataCebo in 2020 with the goal of growing the project. Today, DataCebo is the proud developer of SDV, the largest ecosystem for synthetic data generation & evaluation. It is home to multiple libraries that support synthetic data, including:
- 🔄 Data discovery & transformation. Reverse the transforms to reproduce realistic data.
- 🧠 Multiple machine learning models -- ranging from Copulas to Deep Learning -- to create tabular, multi table and time series data.
- 📊 Measuring quality and privacy of synthetic data, and comparing different synthetic data generation models.
Get started using the SDV package -- a fully integrated solution and your one-stop shop for synthetic data. Or, use the standalone libraries for specific needs.
Owner
- Name: The Synthetic Data Vault Project
- Login: sdv-dev
- Kind: organization
- Email: sdv@sdv.dev
- Website: https://sdv.dev
- Repositories: 9
- Profile: https://github.com/sdv-dev
Committers
Last synced: 9 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Andrew Montanez | a****w@s****v | 345 |
| Carles Sala | c****s@p****m | 278 |
| Felipe Alex Hofmann | f****o@g****m | 152 |
| Manuel Alvarez | m****l@p****m | 146 |
| Plamen Valentinov Kolev | 4****r | 145 |
| JDTheRipperPC | j****c@g****m | 141 |
| Katharine Xiao | 2****o | 91 |
| R-Palazzo | 1****o | 76 |
| Frances Hartwell | f****s@d****m | 74 |
| John La | l****7 | 58 |
| SDV Team | 9****m | 56 |
| amontane | a****w@M****l | 24 |
| Gaurav Sheni | g****i@g****m | 13 |
| github-actions[bot] | 4****] | 10 |
| Neha Patki | n****i@g****m | 9 |
| Roy Wedge | r****e@d****m | 9 |
| Patrick | 3****m | 7 |
| amontane | a****w@d****u | 5 |
| Arash Akhgari | 8****i | 4 |
| amontane | a****w@d****u | 3 |
| amontane | a****w@d****U | 3 |
| amontane | a****w@d****u | 3 |
| amontane | a****w@d****u | 3 |
| Sarah Alnegheimish | 4****h | 3 |
| Aylr | A****r | 2 |
| amontane | a****w@M****t | 2 |
| amontane | a****w@d****U | 2 |
| amontane | a****w@d****U | 2 |
| amontane | a****w@d****u | 2 |
| tssbas | 8****s | 1 |
| and 21 more... | ||
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 749
- Total pull requests: 885
- Average time to close issues: 3 months
- Average time to close pull requests: 5 days
- Total issue authors: 206
- Total pull request authors: 22
- Average comments per issue: 1.44
- Average comments per pull request: 1.11
- Merged pull requests: 715
- Bot issues: 0
- Bot pull requests: 22
Past Year
- Issues: 216
- Pull requests: 355
- Average time to close issues: 19 days
- Average time to close pull requests: 4 days
- Issue authors: 54
- Pull request authors: 14
- Average comments per issue: 0.87
- Average comments per pull request: 1.18
- Merged pull requests: 270
- Bot issues: 0
- Bot pull requests: 4
Top Authors
Issue Authors
- npatki (194)
- amontanez24 (88)
- frances-h (46)
- srinify (34)
- R-Palazzo (30)
- pvk-developer (27)
- gsheni (16)
- fealho (13)
- Ng-ms (10)
- rwedge (9)
- csala (9)
- jalr4ever (8)
- celsofranssa (8)
- lajohn4747 (5)
- wilcovanvorstenbosch (4)
Pull Request Authors
- R-Palazzo (143)
- fealho (136)
- sdv-team (134)
- amontanez24 (100)
- pvk-developer (96)
- lajohn4747 (95)
- frances-h (85)
- gsheni (35)
- rwedge (23)
- github-actions[bot] (20)
- dbrown (2)
- Deathn0t (2)
- dependabot[bot] (2)
- eltociear (2)
- omelyanchikd (2)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 5
-
Total downloads:
- pypi 67,348 last-month
- Total docker downloads: 82,018
-
Total dependent packages: 26
(may contain duplicates) -
Total dependent repositories: 37
(may contain duplicates) - Total versions: 354
- Total maintainers: 11
pypi.org: sdv
Generate synthetic data for single table, multi table and sequential data
- Documentation: https://sdv.readthedocs.io/
- License: BSL-1.1
-
Latest release: 1.26.0
published 6 months ago
Rankings
Maintainers (9)
proxy.golang.org: github.com/sdv-dev/sdv
- Documentation: https://pkg.go.dev/github.com/sdv-dev/sdv#section-documentation
- License: other
-
Latest release: v1.26.0
published 6 months ago
Rankings
proxy.golang.org: github.com/sdv-dev/SDV
- Documentation: https://pkg.go.dev/github.com/sdv-dev/SDV#section-documentation
- License: other
-
Latest release: v1.26.0
published 6 months ago
Rankings
spack.io: py-sdv
The Synthetic Data Vault (SDV) is a Synthetic Data Generation ecosystem of libraries that allows users to easily learn single-table, multi-table and timeseries datasets to later on generate new Synthetic Data that has the same format and statistical properties as the original dataset.
- Homepage: https://github.com/sdv-dev/SDV
- License: []
-
Latest release: 0.14.0
published almost 4 years ago
Rankings
conda-forge.org: sdv
- Homepage: https://github.com/sdv-dev/SDV
- License: BUSL-1.1
-
Latest release: 0.17.1
published over 3 years ago