deepchecks
Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.
Science Score: 54.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
2 of 53 committers (3.8%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (13.5%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.
Basic Info
- Host: GitHub
- Owner: deepchecks
- License: other
- Language: Python
- Default Branch: main
- Homepage: https://docs.deepchecks.com/stable
- Size: 268 MB
Statistics
- Stars: 3,889
- Watchers: 23
- Forks: 277
- Open Issues: 255
- Releases: 0
Topics
Metadata Files
README.md
<!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
<!-- ALL-CONTRIBUTORS-BADGE:END -->
Deepchecks - Continuous Validation for AI & ML: Testing, CI & Monitoring
Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling you to thoroughly test your data and models from research to production.
👋 Join Slack | 📖 Documentation | 🌐 Blog | 🐦 Twitter
🧩 Components
Deepchecks includes: - Deepchecks Testing (Quickstart, docs): - Running built-in & your own custom Checks and Suites for Tabular, NLP & CV validation (open source). - CI & Testing Management (Quickstart, docs): - Collaborating over test results and iterating efficiently until model is production-ready and can be deployed (open source & managed offering). - Deepchecks Monitoring (Quickstart, docs): - Tracking and validating your deployed models behavior when in production (open source & managed offering).
This repo is our main repo as all components use the deepchecks checks in their core. See the Getting Started section for more information about installation and quickstarts for each of the components. If you want to see deepchecks monitoring's code, you can check out the deepchecks/monitoring repo.
⏩ Getting Started
💻 Installation
#### Deepchecks Testing (and CI) Installation
```bash
pip install deepchecks -U --user
```
For installing the nlp / vision submodules or with conda:
- For NLP: Replace ``deepchecks`` with ``"deepchecks[nlp]"``,
and optionally install also``deepchecks[nlp-properties]``
- For Computer Vision: Replace ``deepchecks`` with ``"deepchecks[vision]"``.
- For installing with conda, similarly use: ``conda install -c conda-forge deepchecks``.
Check out the full installation instructions for deepchecks testing [here](https://docs.deepchecks.com/stable/getting-started/installation.html).
#### Deepchecks Monitoring Installation
To use deepchecks for production monitoring, you can either use our SaaS service, or deploy a local instance in one line on Linux/MacOS (Windows is WIP!) with Docker.
Create a new directory for the installation files, open a terminal within that directory and run the following:
```
pip install deepchecks-installer
deepchecks-installer install-monitoring
```
This will automatically download the necessary dependencies, run the installation process
and then start the application locally.
The installation will take a few minutes. Then you can open the deployment url (default is http://localhost),
and start the system onboarding. Check out the full monitoring [open source installation & quickstart](https://docs.deepchecks.com/monitoring/stable/getting-started/deploy_self_host_open_source.html).
Note that the open source product is built such that each deployment supports monitoring of
a single model.
🏃♀️ Quickstarts
Deepchecks Testing Quickstart
Jump right into the respective quickstart docs:
- [Tabular Quickstarts](https://docs.deepchecks.com/stable/tabular/auto_tutorials/quickstarts/index.html)
- [NLP Quickstarts](https://docs.deepchecks.com/stable/nlp/auto_tutorials/quickstarts/index.html)
- [Vision Quickstarts](https://docs.deepchecks.com/stable/vision/auto_tutorials/quickstarts/index.html)
to have it up and running on your data.
Inside the quickstarts, you'll see how to create the relevant deepchecks object for holding your data and metadata
([Dataset](https://docs.deepchecks.com/stable/tabular/usage_guides/dataset_object.html?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=running_a_suite), [TextData](https://docs.deepchecks.com/stable/nlp/usage_guides/text_data_object.html?utm_source=github.com&utm_medium=referral&utm_campaign=readme&utm_content=running_a_suite) or [VisionData](https://docs.deepchecks.com/stable/vision/usage_guides/visiondata_object.html), corresponding to the data type), and run a Suite or Check.
The code snippet for running it will look something like the following, depending on the chosen Suite or Check.
```python
from deepchecks.tabular.suites import model_evaluation
suite = model_evaluation()
suite_result = suite.run(train_dataset=train_dataset, test_dataset=test_dataset, model=model)
suite_result.save_as_html() # replace this with suite_result.show() or suite_result.show_in_window() to see results inline or in window
# or suite_result.results[0].value with the relevant check index to process the check result's values in python
```
The output will be a report that enables you to inspect the status and results of the chosen checks:
Deepchecks Monitoring Quickstart
Jump right into the
[open source monitoring quickstart docs](https://docs.deepchecks.com/monitoring/stable/getting-started/deploy_self_host_open_source.html)
to have it up and running on your data.
You'll then be able to see the checks results over time, set alerts, and interact
with the dynamic deepchecks UI that looks like this:
Deepchecks CI & Testing Management Quickstart
Deepchecks managed CI & Testing management is currently in closed preview.
[Book a demo](https://deepchecks.com/book-demo/) for more information about the offering.
🧮 How does it work?
At its core, deepchecks includes a wide variety of built-in Checks, for testing all types of data and model related issues. These checks are implemented for various models and data types (Tabular, NLP, Vision), and can easily be customized and expanded.
The check results can be used to automatically make informed decisions about your model's production-readiness, and for monitoring it over time in production. The check results can be examined with visual reports (by saving them to an HTML file, or seeing them in Jupyter), processed with code (using their pythonic / json output), and inspected and collaborated on with Deepchecks' dynamic UI (for examining test results and for production monitoring).
✅ Deepchecks' Core: The Checks
- All of the Checks and the framework for customizing them are implemented inside
the Deepchecks Testing Python package (this repo).
- Each check tests for a specific potential problem. Deepchecks has many pre-implemented checks for
finding issues with the model's performance (e.g. identifying weak segments),
data distribution (e.g. detect drifts or leakages) and data integrity (e.g. find conflicting labels).
- Customizable: each check has many configurable parameters, and custom checks can easily be implemented.
- Can be run manually (during research) or triggered automatically (in CI processes or production monitoring)
- Check results can be consumed by:
- Visual output report - [Saving to HTML](https://docs.deepchecks.com/stable/general/usage/export_save_results.html)(``result.save_to_html('output_report_name.html')``) or [viewing them](https://docs.deepchecks.com/stable/general/usage/showing_results.html) in Jupyter (``result.show()``).
- Processing with code - with python using the check result's ``value`` attribute, or saving a
[JSON output](https://docs.deepchecks.com/stable/general/usage/export_save_results.html)
- Deepchecks' UI - for dynamic inspection and collaboration (of test results and production monitoring)
- Optional conditions can be added and customized, to automatically validate check results,
with a a pass ✓, fail ✖ or warning ! status
- An ordered list of checks (with optional conditions) can be run together in a "Suite"
(and the output is a concluding report of all checks that ran)
📜 Open Source vs Paid
Deepchecks' projects (deepchecks/deepchecks & deepchecks/monitoring) are open source and are released under AGPL 3.0.
The only exception are the Deepchecks Monitoring components (in the deepchecks/monitoring repo), that are under the
(backend/deepchecks_monitoring/ee)
directory, that are subject to a commercial license (see the license here).
That directory isn't used by default, and is packaged as part of the deepchecks monitoring repository simply to
support upgrading to the commercial edition without downtime.
Enabling premium features (contained in the backend/deepchecks_monitoring/ee directory) with a self-hosted instance requires a Deepchecks license.
To learn more, book a demo or see our pricing page.
Looking for a 💯% open-source solution for deepcheck monitoring? Check out the Monitoring OSS repository, which is purged of all proprietary code and features.
👭 Community, Contributing, Docs & Support
Deepchecks is an open source solution. We are committed to a transparent development process and highly appreciate any contributions. Whether you are helping us fix bugs, propose new features, improve our documentation or spread the word, we would love to have you as part of our community.
- Give us a ⭐️ github star ⭐️ on the top of this page to support what we're doing, it means a lot for open source projects!
- Read our docs for more info about how to use and customize deepchecks, and for step-by-step tutorials.
- Post a Github Issue to submit a bug report, feature request, or suggest an improvement.
- To contribute to the package, check out our first good issues and contribution guidelines, and open a PR.
Join our Slack to give us feedback, connect with the maintainers and fellow users, ask questions, get help for package usage or contributions, or engage in discussions about ML testing!
✨ Contributors
Thanks goes to these wonderful people (emoji key):
This project follows the all-contributors specification. Contributions of any kind are welcome!
Owner
- Name: deepchecks
- Login: deepchecks
- Kind: organization
- Email: info@deepchecks.com
- Website: www.deepchecks.com
- Repositories: 9
- Profile: https://github.com/deepchecks
Citation (CITATION.cff)
cff-version: 1.2.0
title: "Deepchecks: A Library for Testing and Validating Machine Learning Models and Data"
message: "If you use this software, please cite it using the metadata from this file."
type: software
authors:
- family-names: Chorev
given-names: Shir
email: shir@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Tannor
given-names: Philip
email: philip@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Ben Israel
given-names: Dan
email: danb@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Bressler
given-names: Noam
email: noam@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Gabbay
given-names: Itay
email: itay@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Hutnik
given-names: Nir
email: nir@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Liberman
given-names: Jonatan
email: jonatan@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Perlmutter
given-names: Matan
email: matan@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Romanyshyn
given-names: Yurii
email: yurii@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Rokach
given-names: Lior
email: liorrk@bgu.ac.il
affiliation: Deepchecks Ltd. and Department of Software and Info. Sys. Eng. Ben-Gurion University of the Negev
url: "https://github.com/deepchecks/deepchecks"
preferred-citation:
type: article
authors:
- family-names: Chorev
given-names: Shir
email: shir@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Tannor
given-names: Philip
email: philip@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Ben Israel
given-names: Dan
email: danb@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Bressler
given-names: Noam
email: noam@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Gabbay
given-names: Itay
email: itay@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Hutnik
given-names: Nir
email: nir@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Liberman
given-names: Jonatan
email: jonatan@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Perlmutter
given-names: Matan
email: matan@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Romanyshyn
given-names: Yurii
email: yurii@deepchecks.com
affiliation: Deepchecks Ltd.
- family-names: Rokach
given-names: Lior
email: liorrk@bgu.ac.il
affiliation: Deepchecks Ltd. and Department of Software and Info. Sys. Eng. Ben-Gurion University of the Negev
title: "Deepchecks: A Library for Testing and Validating Machine Learning Models and Data"
journal: Journal of Machine Learning Research
year: 2022
volume: 23
number: 265
start: 1
end: 6
url: "http://jmlr.org/papers/v23/22-0281.html"
GitHub Events
Total
- Create event: 20
- Release event: 3
- Issues event: 15
- Watch event: 279
- Delete event: 4
- Member event: 1
- Issue comment event: 23
- Push event: 122
- Pull request review comment event: 4
- Pull request review event: 21
- Pull request event: 74
- Fork event: 25
Last Year
- Create event: 20
- Release event: 3
- Issues event: 15
- Watch event: 279
- Delete event: 4
- Member event: 1
- Issue comment event: 23
- Push event: 122
- Pull request review comment event: 4
- Pull request review event: 21
- Pull request event: 74
- Fork event: 25
Committers
Last synced: 9 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Itay Gabbay | i****y@d****m | 213 |
| Noam Bressler | n****r@g****m | 211 |
| matanper | m****n@d****m | 190 |
| JKL98ISR | j****b@g****m | 188 |
| Yurii Romanyshyn | 7****n | 111 |
| Nir Hutnik | 9****k | 93 |
| Nadav Barak | 6****k | 90 |
| DBI | 4****n | 83 |
| shir22 | 3****2 | 67 |
| matanper | m****r@g****m | 48 |
| Sol | 9****Y | 31 |
| DanArlowski | 5****i | 27 |
| allcontributors[bot] | 4****] | 25 |
| deepchecks-bot | 9****t | 19 |
| github-actions[bot] | g****] | 11 |
| Harsh Jain | 1****s | 8 |
| Shiv Shankar Dayal | s****l@g****m | 8 |
| Dan Arlowski | d****n@d****m | 7 |
| DanBasson | d****7@g****m | 6 |
| Harsh Jain | h****4@g****m | 6 |
| S.Kishore | s****v@g****m | 6 |
| Nir Hutnik | n****r@d****m | 4 |
| Cemal GURPINAR | 3****r | 2 |
| Michael Marien | m****h@g****m | 2 |
| Ofir Nissim | E****r@g****m | 2 |
| Shay Palachy | s****5 | 2 |
| shiritdvir | 5****r | 2 |
| arterm-sedov | 6****v | 1 |
| Shay Tsadok | s****7@g****m | 1 |
| RonItay | 3****y | 1 |
| and 23 more... | ||
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 81
- Total pull requests: 304
- Average time to close issues: about 2 months
- Average time to close pull requests: about 1 month
- Total issue authors: 39
- Total pull request authors: 28
- Average comments per issue: 1.22
- Average comments per pull request: 0.28
- Merged pull requests: 204
- Bot issues: 0
- Bot pull requests: 11
Past Year
- Issues: 10
- Pull requests: 26
- Average time to close issues: 5 days
- Average time to close pull requests: 3 days
- Issue authors: 10
- Pull request authors: 7
- Average comments per issue: 0.7
- Average comments per pull request: 0.31
- Merged pull requests: 17
- Bot issues: 0
- Bot pull requests: 2
Top Authors
Issue Authors
- Nadav-Barak (16)
- nirhutnik (14)
- noamzbr (8)
- hjain5164 (2)
- bgalvao (2)
- j-adamczyk (2)
- PushaBe (1)
- borisRa (1)
- milkajakovljevic (1)
- a-recknagel (1)
- srraht (1)
- arterm-sedov (1)
- ItayGabbay (1)
- pr3sh (1)
- PleaseNotAnotherUsername (1)
Pull Request Authors
- ItayGabbay (98)
- noamzbr (61)
- JKL98ISR (37)
- Nadav-Barak (36)
- nirhutnik (24)
- yromanyshyn (15)
- harsh-deepchecks (13)
- shir22 (12)
- allcontributors[bot] (10)
- matanper (7)
- dependabot[bot] (6)
- hjain5164 (5)
- kishore-s-15 (3)
- RayanAAY-ops (3)
- DanielAvdar (2)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 3
-
Total downloads:
- pypi 37,914 last-month
- Total docker downloads: 145
-
Total dependent packages: 7
(may contain duplicates) -
Total dependent repositories: 92
(may contain duplicates) - Total versions: 79
- Total maintainers: 2
pypi.org: deepchecks
Package for validating your machine learning model and data
- Homepage: https://github.com/deepchecks/deepchecks
- Documentation: https://docs.deepchecks.com
- License: GNU Affero General Public License v3 or later (AGPLv3+)
-
Latest release: 0.19.1
published about 1 year ago
Rankings
Maintainers (1)
pypi.org: runml-checks
Package for validating your machine learning model and data
- Homepage: https://github.com/deepchecks/deepchecks
- Documentation: https://docs.deepchecks.com
- License: GNU Affero General Public License v3 or later (AGPLv3+)
-
Latest release: 1.0.0
published over 3 years ago
Rankings
Maintainers (1)
conda-forge.org: deepchecks
- Homepage: https://github.com/deepchecks/deepchecks
- License: AGPL-3.0-or-later
-
Latest release: 0.9.2
published over 3 years ago
Rankings
Dependencies
- actions/checkout v3 composite
- actions/setup-python v3 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- pilosus/action-pip-license-checker v0.6.1 composite
- actions/checkout v2 composite
- github/codeql-action/analyze v1 composite
- github/codeql-action/autobuild v1 composite
- github/codeql-action/init v1 composite
- actions/checkout v1 composite
- conda-incubator/setup-miniconda v2 composite
- maxibor/conda-package-publish-action v1.1 composite
- actions/checkout v2 composite
- actions/setup-python v2 composite
- actions/checkout v2 composite
- actions/download-artifact v2 composite
- actions/setup-python v2 composite
- actions/upload-artifact v2 composite
- andymckay/labeler 1.0.4 composite
- actions/checkout v2 composite
- actions/setup-python v2 composite
- jamescurtin/isort-action master composite
- mheap/github-action-required-labels v1 composite
- actions/checkout v2 composite
- actions/setup-python v2 composite
- marvinpinto/action-automatic-releases latest composite
- docutils *
- nbsphinx >=0.8.7
- numpydoc >=1.1.0
- pydata-sphinx-theme >=0.7.2,<0.9.0
- pypandoc >=1.7.2
- readthedocs-sphinx-search *
- sphinx ==4.5.0
- sphinx-copybutton >=0.4.0
- sphinx-gallery >=0.10.1
- sphinx-reredirects >=0.0.1
- sphinxcontrib-applehelp >=1.0.2
- sphinxcontrib-devhelp >=1.0.2
- sphinxcontrib-htmlhelp >=2.0.0
- sphinxcontrib-jsmath >=1.0.1
- sphinxcontrib-qthelp >=1.0.3
- sphinxcontrib-serializinghtml >=1.1.5
- boto3 * test
- catboost * test
- fsspec * test
- joblib * test
- s3fs * test
- xgboost * test
- Pillow >=7.1.2 development
- PyYAML >=5.3.1 development
- beautifulsoup4 >=4.11.1 development
- catboost * development
- coveralls * development
- deepdiff * development
- flake8 ==4.0.1 development
- flake8-eradicate * development
- flake8-rst * development
- flake8-spellcheck * development
- gower ==0.0.5 development
- isort * development
- jupyter * development
- jupyterlab * development
- kaleido * development
- lightgbm * development
- matplotlib >=3.2.2 development
- nbval * development
- notebook * development
- numpy >=1.18.5 development
- opencv-python >=4.1.2 development
- pandas ==1.3.5 development
- protobuf >=3.12.0,<4.0dev development
- pydocstyle * development
- pyhamcrest * development
- pylint ==2.13.5 development
- pytest * development
- pytest-cov * development
- requests >=2.23.0 development
- scikit-learn ==1.0.2 development
- scipy >=1.4.1 development
- seaborn >=0.11.0 development
- tox * development
- tqdm >=4.41.0 development
- twine * development
- wandb >=0.12.15,<0.13.0 development
- xgboost * development
- PyNomaly >=0.3.3
- beautifulsoup4 >=4.11.1
- category-encoders >=2.3.0
- dataclasses >=0.6
- importlib_metadata >=1.4
- importlib_resources >=1.3
- ipykernel >=4.10.1
- ipykernel >=5.3.0
- ipython >=7.15.0,<8
- ipython >=5.5.0,<8
- ipywidgets >=7.6.5,<8
- ipywidgets >=7.5.0,<8
- jsonpickle >=2
- matplotlib >=3.3.4
- numpy >=1.19
- pandas >=1.1.5
- plotly >=5.8.0
- pyzmq <24.0.0
- scikit-learn >=0.23.2
- scipy >=1.4.1
- statsmodels >=0.13.5
- statsmodels >=0.11.0
- tqdm >=4.62.3
- typing_extensions >=4.0.0
- albumentations >=1.1.0
- imagehash >=4.0.0
- imgaug >=0.4.0
- lxml >=4.0.0
- opencv-python >=4.5.5.62
- pytorch-ignite >=0.4.8
- requests >=2.22.0
- seaborn >=0.1.0
