ck
Collective Knowledge (CK), Collective Mind (CM/CMX) and MLPerf automations: community-driven projects to facilitate collaborative and reproducible research and to learn how to run AI, ML, and other emerging workloads more efficiently and cost-effectively across diverse models, datasets, software, and hardware using MLPerf methodology and benchmarks
Science Score: 77.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
✓DOI references
Found 3 DOI reference(s) in README -
✓Academic publication links
Links to: arxiv.org, acm.org -
✓Committers with academic emails
2 of 32 committers (6.3%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.9%) to scientific vocabulary
Keywords
Repository
Collective Knowledge (CK), Collective Mind (CM/CMX) and MLPerf automations: community-driven projects to facilitate collaborative and reproducible research and to learn how to run AI, ML, and other emerging workloads more efficiently and cost-effectively across diverse models, datasets, software, and hardware using MLPerf methodology and benchmarks
Basic Info
- Host: GitHub
- Owner: mlcommons
- License: apache-2.0
- Language: Python
- Default Branch: master
- Homepage: https://access.cKnowledge.org
- Size: 35.1 MB
Statistics
- Stars: 629
- Watchers: 49
- Forks: 118
- Open Issues: 9
- Releases: 139
Topics
Metadata Files
README.md
Collective Knowledge project (CK)
Collective Knowledge (CK) is a community-driven project dedicated to supporting open science, enhancing reproducible research, and fostering collaborative learning on how to run AI, ML, and other emerging workloads in the most efficient and cost-effective way across diverse models, data sets, software and hardware: [ white paper ].
It includes the following sub-projects.
Collective Mind project (MLCommons CM)
The Collective Mind automation framework (CM) was developed to support open science and facilitate collaborative, reproducible, and reusable research, development, and experimentation based on FAIR principles.
It helps users non-intrusively convert their software projects into file-based repositories of portable and reusable artifacts (code, data, models, scripts) with extensible metadata and reusable automations, a unified command-line interface, and a simple Python API.
Such artifacts can be easily chained together into portable and technology-agnostic automation workflows, enabling users to rerun, reproduce, and reuse complex experimental setups across diverse and rapidly evolving models, datasets, software, and hardware.
For example, CM helps to modularize, automate and customize MLPerf benchmarks.
Legacy CM API and CLI (2021-2024)
See the project page for more details.
Legacy and simplified CM and MLPerf automations were donated to MLCommons by Grigori Fursin, the cTuning foundation and OctoML. They are now supported by the MLCommons Infra WG (MLCFlow, MLC scripts, mlcr ...).
New CM API and CLI (CMX, 2025+)
Collective Mind eXtension or Common Metadata eXchange (CMX) is the next evolution of the Collective Mind automation framework (MLCommons CM) designed to enhance simplicity, flexibility, and extensibility of automations based on user feedback. It is backwards compatible with CM, released along with CM in the cmind package and can serve as drop-in replacement for CM and legacy MLPerf automations while providing a simpler and more robust interface.
See the project page and CMX4MLOps automations for more details.
MLOps and MLPerf automations
We have developed a collection of portable, extensible and technology-agnostic automation recipes with a common CLI and Python API (CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications on diverse platforms with any software and hardware.
The two key automations are script and cache: see online catalog at CK playground, online MLCommons catalog.
CM scripts extend the concept of cmake with simple Python automations, native scripts
and JSON/YAML meta descriptions. They require Python 3.8+ with minimal dependencies and are
continuously extended by the community and MLCommons members
to run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux
and any other operating system, in a cloud or inside automatically generated containers
while keeping backward compatibility.
See the online MLPerf documentation
at MLCommons to run MLPerf inference benchmarks across diverse systems using CMX.
Just install pip install cmx4mlperf and substitute the following commands and flags:
* cm -> cmx
* mlc -> cmlc
* mlcr -> cmlcr
* -v -> --v
Collective Knowledge Playground
Collective Knowledge Playground - a unified and open-source platform designed to index all CM/CMX automations similar to PYPI and assist users in preparing CM/CMX commands to:
- aggregate, process, visualize, and compare MLPerf benchmarking results for AI and ML systems
- run MLPerf benchmarks
- organize open and reproducible optimization challenges and tournaments.
Artifact Evaluation and Reproducibility Initiatives
Artifact Evaluation automation - a community-driven initiative leveraging CK, CM and CMX to automate artifact evaluation and support reproducibility efforts at ML and systems conferences.
Legacy projects
- CM-MLOps (2021)
- CM4MLOps (2022-2024)
- CK automation framework v1 and v2
License
Copyright
Copyright (c) 2021-2025 MLCommons
Grigori Fursin, the cTuning foundation and OctoML donated this project to MLCommons to benefit everyone.
Copyright (c) 2014-2021 cTuning foundation
Author
Maintainers
- Legacy CM, CM4MLOps and MLPerf automations: MLCommons infra WG
- CMX (the next generation of CM since 2025): Grigori Fursin
Concepts
To learn more about the motivation behind this project, please explore the following articles and presentations:
- HPCA'25 article "MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI": [ Arxiv ], [ tutorial to reproduce results using CM/CMX ]
- NeuralMagic's vLLM MLPerf inference 4.1 submission automated by CM: [README]
- SDXL MLPerf inference 4.1 submission automated by CM: [README]
- "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ ArXiv ]
- ACM REP'23 keynote about the MLCommons CM automation framework: [ slides ]
- ACM TechTalk'21 about Collective Knowledge project: [ YouTube ] [ slides ]
- Journal of Royal Society'20: [ paper ]
Acknowledgments
This open-source project was created by Grigori Fursin and sponsored by cTuning.org, OctoAI and HiPEAC. Grigori donated this project to MLCommons to modularize and automate MLPerf benchmarks, benefit the community, and foster its development as a collaborative, community-driven effort.
We thank MLCommons, FlexAI and cTuning for supporting this project, as well as our dedicated volunteers and collaborators for their feedback and contributions!
If you found the CM, CMX and MLPerf automations helpful, kindly reference this article: [ ArXiv ], [ BibTex ].
You are welcome to contact the author to discuss long-term plans and potential collaboration.
Owner
- Name: MLCommons
- Login: mlcommons
- Kind: organization
- Repositories: 79
- Profile: https://github.com/mlcommons
Citation (citation.bib)
@misc{fursin2024enabling,
title={Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments},
author={Grigori Fursin},
year={2024},
eprint={2406.16791},
archivePrefix={arXiv},
primaryClass={id='cs.LG' full_name='Machine Learning' is_active=True alt_name=None in_archive='cs' is_general=False description='Papers on all aspects of machine learning research (supervised, unsupervised, reinforcement learning, bandit problems, and so on) including also robustness, explanation, fairness, and methodology. cs.LG is also an appropriate primary category for applications of machine learning methods.'}
}
@article{doi:10.1098/rsta.2020.0211,
author = {Fursin, Grigori},
title = {Collective knowledge: organizing research projects as a database of reusable components and portable workflows with common interfaces},
journal = {Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences},
volume = {379},
number = {2197},
pages = {20200211},
year = {2021},
doi = {10.1098/rsta.2020.0211},
URL = {https://royalsocietypublishing.org/doi/abs/10.1098/rsta.2020.0211},
eprint = {https://royalsocietypublishing.org/doi/pdf/10.1098/rsta.2020.0211},
howpublished = {https://doi.org/10.1098/rsta.2020.0211}
}
@misc{acm_rep_23_cm_keynote,
author = {Fursin, Grigori},
year = "2023",
booktitle = "{Keynote at the 1st ACM conference on reproducibility and replicability (ACM REP'23)}",
title = "Collective Mind: toward a common language to facilitate reproducible research and technology transfer",
howpublished = {https://doi.org/10.5281/zenodo.8105339}
}
GitHub Events
Total
- Create event: 26
- Issues event: 44
- Release event: 16
- Watch event: 32
- Delete event: 7
- Issue comment event: 104
- Push event: 101
- Pull request review event: 51
- Pull request event: 128
- Fork event: 8
Last Year
- Create event: 26
- Issues event: 44
- Release event: 16
- Watch event: 32
- Delete event: 7
- Issue comment event: 104
- Push event: 101
- Pull request review event: 51
- Pull request event: 128
- Fork event: 8
Committers
Last synced: almost 3 years ago
All Time
- Total Commits: 3,737
- Total Committers: 32
- Avg Commits per committer: 116.781
- Development Distribution Score (DDS): 0.581
Top Committers
| Name | Commits | |
|---|---|---|
| Grigori Fursin | G****n@c****g | 1,567 |
| Arjun Suresh | a****7@g****m | 1,365 |
| Grigori Fursin | g****n@c****g | 501 |
| Grigori Fursin | g****n@g****m | 103 |
| Grigori Fursin | g****n@u****m | 68 |
| dsavenko | d****o@x****m | 40 |
| Grigori Fursin | G****n@C****i | 15 |
| anandhu-eng | a****s@g****m | 13 |
| Anton Lokhmotov | a****n@d****m | 12 |
| Thomas Zhu | t****h@h****m | 8 |
| Grigori Fursin | c****x@c****t | 5 |
| Grigori Fursin | G****n@c****i | 5 |
| Leo Gordon | e****4@u****m | 4 |
| Anton Lokhmotov | a****v@g****m | 4 |
| Ailurus1 | i****5@g****m | 4 |
| Grigori Fursin | g****n@c****i | 3 |
| Ailurus1 | k****2@y****u | 2 |
| Grigori Fursin | g****i@o****i | 2 |
| Grigori Fursin | f****e@h****m | 2 |
| Himanshu Dutta | m****a@g****m | 2 |
| Alexey Kravets | a****s@a****m | 1 |
| Bruno Ferreira | b****f@g****m | 1 |
| Dave Greasley | D****y@u****m | 1 |
| Ailurus1 | k****2@y****u | 1 |
| alered01 | 4****1@u****m | 1 |
| xintin | g****a@c****u | 1 |
| Ubuntu | u****u@i****l | 1 |
| Lahiru Rasnayake | l****e@n****o | 1 |
| Grigori Fursin | f****n@c****e | 1 |
| Ilya Kozulin | 7****1@u****m | 1 |
| and 2 more... | ||
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 4 months ago
All Time
- Total issues: 194
- Total pull requests: 552
- Average time to close issues: 4 months
- Average time to close pull requests: about 10 hours
- Total issue authors: 40
- Total pull request authors: 20
- Average comments per issue: 2.79
- Average comments per pull request: 1.17
- Merged pull requests: 505
- Bot issues: 0
- Bot pull requests: 23
Past Year
- Issues: 34
- Pull requests: 126
- Average time to close issues: 7 days
- Average time to close pull requests: about 5 hours
- Issue authors: 3
- Pull request authors: 7
- Average comments per issue: 1.03
- Average comments per pull request: 1.17
- Merged pull requests: 110
- Bot issues: 0
- Bot pull requests: 5
Top Authors
Issue Authors
- gfursin (105)
- arjunsuresh (28)
- anandhu-eng (6)
- KingICCrab (5)
- WarrenSchultz (4)
- willamloo3192 (4)
- jdesfossez (2)
- sahilavaran (2)
- JoachimMoe (2)
- dmgusev (1)
- Sachin-23 (1)
- Seemapatil1998 (1)
- Agalakdak (1)
- keithachorn-intel (1)
- Adityashaw (1)
Pull Request Authors
- gfursin (344)
- arjunsuresh (279)
- dependabot[bot] (32)
- ctuning-admin (26)
- anandhu-eng (8)
- Ailurus1 (7)
- jdesfossez (3)
- pnfox (2)
- Chaitanya110703 (2)
- Dnaynu (2)
- himanshu-dutta (2)
- nathanw-mlc (1)
- Maximallnyi (1)
- SennikovAndrey (1)
- morphine00 (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 8
-
Total downloads:
- pypi 7,422 last-month
- Total docker downloads: 6,069
-
Total dependent packages: 0
(may contain duplicates) -
Total dependent repositories: 27
(may contain duplicates) - Total versions: 217
- Total maintainers: 1
pypi.org: ck
Collective Knowledge - a lightweight knowledge manager to organize, cross-link, share and reuse artifacts and workflows based on FAIR principles
- Homepage: https://github.com/mlcommons/ck
- Documentation: https://ck.readthedocs.io/
- License: Apache 2.0
-
Latest release: 2.6.4
published over 1 year ago
Rankings
Maintainers (1)
pypi.org: cmind
Common Metadata eXchange framework (CMX) and Collective Mind automation framework (CM)
- Homepage: https://cKnowledge.org
- Documentation: https://docs.mlcommons.org/ck
- License: Apache 2.0
-
Latest release: 4.2.0
published 8 months ago
Rankings
Maintainers (1)
pypi.org: cmind4mlperf
TBD
- Homepage: https://github.com/mlcommons/ck
- Documentation: https://cmind4mlperf.readthedocs.io/
- License: Apache Software License (Apache 2.0)
-
Latest release: 0.0.0.2
published over 1 year ago
Rankings
Maintainers (1)
pypi.org: cmx4mlperf
CMX4MLPerf repository with legacy MLPerf automations
- Homepage: https://github.com/mlcommons/ck
- Documentation: https://docs.mlcommons.org/ck
- License: Apache 2.0
-
Latest release: 1.0.0
published 10 months ago
Rankings
Maintainers (1)
pypi.org: cmx4mlops
CMX4MLOps repository
- Homepage: https://github.com/mlcommons/ck
- Documentation: https://cmx4mlops.readthedocs.io/
- License: Apache 2.0
-
Latest release: 1.0.1
published 11 months ago
Rankings
Maintainers (1)
pypi.org: cmind4system
TBD
- Homepage: https://github.com/mlcommons/ck
- Documentation: https://cmind4system.readthedocs.io/
- License: Apache Software License (Apache 2.0)
-
Latest release: 0.0.0.1
published over 1 year ago
Rankings
Maintainers (1)
pypi.org: cm4system
TBD
- Homepage: https://github.com/mlcommons/ck
- Documentation: https://cm4system.readthedocs.io/
- License: Apache Software License (Apache 2.0)
-
Latest release: 0.0.0.1
published over 1 year ago
Rankings
Maintainers (1)
pypi.org: cm4research
TBD
- Homepage: https://github.com/mlcommons/ck
- Documentation: https://cm4research.readthedocs.io/
- License: Apache Software License (Apache 2.0)
- Status: removed
-
Latest release: 0.0.0.1
published over 1 year ago
Rankings
Maintainers (1)
Dependencies
- mlcommons/cla-bot master composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- ck *
- click >=7.0
- requests *
- virtualenv *
- ck *
- pyyaml *
- pyyaml *
- requests *
- pyyaml *
- requests *
- pyyaml *
- requests *
- cmind >=1.0.3
- requests *
- Pillow *
- numpy *
- opencv-python *
- Pillow *
- numpy *
- requests *
- attrs *
- decorator *
- matplotlib *
- onnx *
- opencv-python *
- psutil *
- scipy *
- giturlparse *
- numpy *
- pandas *
- requests *
- wheel *
- pyyaml *