https://github.com/modelscope/data-juicer
Data processing for and with foundation models! π π π½ β‘οΈ β‘οΈπΈ πΉ π·
Science Score: 46.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
βCITATION.cff file
-
βcodemeta.json file
Found codemeta.json file -
β.zenodo.json file
Found .zenodo.json file -
βDOI references
-
βAcademic publication links
Links to: arxiv.org, ieee.org -
βCommitters with academic emails
2 of 33 committers (6.1%) from academic institutions -
βInstitutional organization owner
-
βJOSS paper metadata
-
βScientific vocabulary similarity
Low similarity (8.3%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
Data processing for and with foundation models! π π π½ β‘οΈ β‘οΈπΈ πΉ π·
Basic Info
- Host: GitHub
- Owner: modelscope
- License: apache-2.0
- Language: Python
- Default Branch: main
- Homepage: https://modelscope.github.io/data-juicer/
- Size: 453 MB
Statistics
- Stars: 5,142
- Watchers: 20
- Forks: 267
- Open Issues: 68
- Releases: 19
Topics
Metadata Files
README.md
[δΈζδΈ»ι‘΅] | [DJ-Cookbook] | [OperatorZoo] | [API] | [Awesome LLM Data]
Data Processing for and with Foundation Models
-B31B1B?logo=arxiv&logoColor=red){kind=icon}
Data-Juicer is a one-stop system to process text and multimodal data for and with foundation models (typically LLMs). We provide a playground with a managed JupyterLab. Try Data-Juicer straight away in your browser! If you find Data-Juicer useful for your research or development, please kindly support us by starting it (then be instantly notified of our new releases) and citing our works.
Platform for AI of Alibaba Cloud (PAI) has deeply integrated Data-Juicer into its data processing products. PAI is an AI Native large model and AIGC engineering platform that provides dataset management, computing power management, model tool chain, model development, model training, model deployment, and AI asset management. For documentation on data processing, please refer to: PAI-Data Processing for Large Models.
Data-Juicer is being actively updated and maintained. We will periodically enhance and add more features, data recipes and datasets. We welcome you to join us, in promoting data-model co-development along with research and applications of foundation models!
[Demo Video] DataJuicer-Agent: Quick start your data processing journey!
https://github.com/user-attachments/assets/6eb726b7-6054-4b0c-905e-506b2b9c7927
[Demo Video] DataJuicer-Sandbox: Better data-model co-dev at a lower cost!
https://github.com/user-attachments/assets/a45f0eee-0f0e-4ffe-9a42-d9a55370089d
News
- π οΈ [2025-06-04] How to process feedback data in the "era of experience"? We propose Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of LLMs, which leverages Data-Juicer for its data pipelines tailored for RFT scenarios.
- π [2025-06-04] Our Data-Model Co-development Survey has been accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)! Welcome to explore and contribute the awesome-list.
- π [2025-06-04] We introduce DetailMaster: Can Your Text-to-Image Model Handle Long Prompts? A synthetic benchmark revealing notable performance drops despite large models' proficiency with short descriptions.
- π [2025-05-06] Our work of Data-Juicer Sandbox has been accepted as a ICML'25 Spotlight (top 2.6% of all submissions)!
- π‘ [2025-03-13] We propose MindGYM: What Matters in Question Synthesis for Thinking-Centric Fine-Tuning? A new data synthesis method that enables large models to self-synthesize high-quality, low-variance data for efficient fine-tuning, (e.g., 16% gain on MathVision using only 400 samples).
- π€ [2025-02-28] DJ has been integrated in Ray's official Ecosystem and Example Gallery. Besides, our patch in DJ2.0 for the streaming JSON reader has been officially integrated by Apache Arrow.
- π [2025-02-27] Our work on contrastive data synthesis, ImgDiff, has been accepted by CVPR'25!
- π‘ [2025-02-05] We propose a new data selection method, Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data. It is theoretically informed, via treating diversity as a reward, achieves better overall performance across 7 benchmarks when post-training SOTA LLMs.
- π [2025-01-11] We release our 2.0 paper, Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for Foundation Models. It now can process 70B data samples within 2.1h, using 6400 CPU cores on 50 Ray nodes from Alibaba Cloud cluster, and deduplicate 5TB data within 2.8h using 1280 CPU cores on 8 Ray nodes.
- π οΈ [2025-01-03] We support post-tuning scenarios better, via 20+ related new OPs, and via unified dataset format compatible to LLaMA-Factory and ModelScope-Swift.
History News:
> - [2024-12-17] We propose *HumanVBench*, which comprises 16 human-centric tasks with synthetic data, benchmarking 22 video-MLLMs' capabilities from views of inner emotion and outer manifestations. See more details in our [paper](https://arxiv.org/abs/2412.17574), and try to [evaluate](https://github.com/modelscope/data-juicer/tree/HumanVBench) your models with it. - [2024-11-22] We release DJ [v1.0.0](https://github.com/modelscope/data-juicer/releases/tag/v1.0.0), in which we refactored Data-Juicer's *Operator*, *Dataset*, *Sandbox* and many other modules for better usability, such as supporting fault-tolerant, FastAPI and adaptive resource management. - [2024-08-25] We give a [tutorial](https://modelscope.github.io/data-juicer/_static/tutorial_kdd24.html) about data processing for multimodal LLMs in KDD'2024. - [2024-08-09] We propose Img-Diff, which enhances the performance of multimodal large language models through *contrastive data synthesis*, achieving a score that is 12 points higher than GPT-4V on the [MMVP benchmark](https://tsb0601.github.io/mmvp_blog/). See more details in our [paper](https://arxiv.org/abs/2408.04594), and download the dataset from [huggingface](https://huggingface.co/datasets/datajuicer/Img-Diff) and [modelscope](https://modelscope.cn/datasets/Data-Juicer/Img-Diff). - [2024-07-24] "Tianchi Better Synth Data Synthesis Competition for Multimodal Large Models" β Our 4th data-centric LLM competition has kicked off! Please visit the competition's [official website](https://tianchi.aliyun.com/competition/entrance/532251) for more information. - [2024-07-17] We utilized the Data-Juicer [Sandbox Laboratory Suite](https://github.com/modelscope/data-juicer/blob/main/docs/Sandbox.md) to systematically optimize data and models through a co-development workflow between data and models, achieving a new top spot on the [VBench](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard) text-to-video leaderboard. The related achievements have been compiled and published in a [paper](http://arxiv.org/abs/2407.11784), and the model has been released on the [ModelScope](https://modelscope.cn/models/Data-Juicer/Data-Juicer-T2V) and [HuggingFace](https://huggingface.co/datajuicer/Data-Juicer-T2V) platforms. - [2024-07-12] Our *awesome list of MLLM-Data* has evolved into a systemic [survey](https://arxiv.org/abs/2407.08583) from model-data co-development perspective. Welcome to [explore](docs/awesome_llm_data.md) and contribute! - [2024-06-01] ModelScope-Sora "Data Directors" creative sprintβOur third data-centric LLM competition has kicked off! Please visit the competition's [official website](https://tianchi.aliyun.com/competition/entrance/532219) for more information. - [2024-03-07] We release **Data-Juicer [v0.2.0](https://github.com/modelscope/data-juicer/releases/tag/v0.2.0)** now! In this new version, we support more features for **multimodal data (including video now)**, and introduce **[DJ-SORA](docs/DJ_SORA.md)** to provide open large-scale, high-quality datasets for SORA-like models. - [2024-02-20] We have actively maintained an *awesome list of LLM-Data*, welcome to [visit](docs/awesome_llm_data.md) and contribute! - [2024-02-05] Our paper has been accepted by SIGMOD'24 industrial track! - [2024-01-10] Discover new horizons in "Data Mixture"βOur second data-centric LLM competition has kicked off! Please visit the competition's [official website](https://tianchi.aliyun.com/competition/entrance/532174) for more information. - [2024-01-05] We release **Data-Juicer v0.1.3** now! In this new version, we support **more Python versions** (3.8-3.10), and support **multimodal** dataset [converting](tools/fmt_conversion/multimodal/README.md)/[processing](docs/Operators.md) (Including texts, images, and audios. More modalities will be supported in the future). Besides, our paper is also updated to [v3](https://arxiv.org/abs/2309.02033). - [2023-10-13] Our first data-centric LLM competition begins! Please visit the competition's official websites, FT-Data Ranker ([1B Track](https://tianchi.aliyun.com/competition/entrance/532157), [7B Track](https://tianchi.aliyun.com/competition/entrance/532158)), for more information.Why Data-Juicer?
Systematic & Reusable: Empowering users with a systematic library of 100+ core OPs, and 50+ reusable config recipes and dedicated toolkits, designed to function independently of specific multimodal LLM datasets and processing pipelines. Supporting data analysis, cleaning, and synthesis in pre-training, post-tuning, en, zh, and more scenarios.
User-Friendly & Extensible: Designed for simplicity and flexibility, with easy-start guides, and DJ-Cookbook containing fruitful demo usages. Feel free to implement your own OPs for customizable data processing.
Data-Juicer now uses AI to automatically rewrite and optimize operator docstrings, generating detailed operator documentation to help users quickly understand the functionality and usage of each operator.
For details about the implementation of this documentation enhancement workflow, please visit the demos/opdocenhanceworkflow folder under the `djagents` branch.
Efficient & Robust: Providing performance-optimized parallel data processing (Aliyun-PAI\Ray\CUDA\OP Fusion), faster with less resource usage, verified in large-scale production environments.
Effect-Proven & Sandbox: Supporting data-model co-development, enabling rapid iteration through the sandbox laboratory, and providing features such as feedback loops and visualization, so that you can better understand and improve your data and models. Many effect-proven datasets and models have been derived from DJ, in scenarios such as pre-training, text-to-video and image-to-text generation.
Doucmentation
- Tutorial
- Useful documents
- Demos
- Tools
- Distributed Fuzzy Deduplication Tools
- Auto Evaluation Toolkit
- GPT EVAL: Evaluate your model with OpenAI API
- Evaluation Results Recorder
- Format Conversion Tools
- Multimodal Tools
- Post Tuning Tools
- Hyper-parameter Optimization for Data Recipe
- Label Studio Service Utility
- Metrics for video generation
- Postprocess Tools
- Preprocess Tools
- Data Scoring
- Third-party
License
Data-Juicer is released under Apache License 2.0.
Contribution and Acknowledgements
Data-Juicer has benefited greatly from and continues to welcome contributions at all levels: new operators (from simple functions to advanced algorithms based on existing papers), data-recipes & processing scenarios, feature requests, efficiency enhancements, bug fixes, better documentation and usage feedback. Please refer to our Developer Guide to get started. Spreading the word in the community and giving the repository a star β are also invaluable forms of support!
Our sincere gratitude goes to all our code contributors who are the cornerstone of this project. We strive to keep the list below updated and look forward to including more names (alphabetical order); please reach out if we have missed any acknowledgements. - Initiated by: Alibaba Tongyi Lab - Co-developed and Optimized with: Alibaba Cloud PAI, Anyscale (Ray Team), Sun Yat-sen University (Knowledge Engineering Lab), NVIDIA (NeMo Team), ... - Used by & Valuable Feedback from: AgentScope, Alibaba Group, Ant Group, BYD Auto, Bytedance, CAS, DiffSynth-Studio, EasyAnimate, Eval-Scope, JD.com, LLaMA-Factory, Nanjing University, OPPO, Peking University, RM-Gallery, RUC, Tsinghua University, Trinity-RFT, UCAS, Xiaohongshu, Xiaomi, Ximalaya, Zhejiang University, ... - Inspired by: Data-Juicer also thanks pioneering open-source projects such as Apache Arrow, BLOOM, RedPajama-Data, Ray, Hugging Face Datasets, ...
We look forward to your feedback and collaboration, including partnership inquiries or proposals for new sub-projects related to Data-Juicer. Feel free to contact via issues, PRs, Slack channel, DingDing group, and e-mails.
References
If you find Data-Juicer useful for your research or development, please kindly cite the following works, 1.0paper, 2.0paper. ``` @inproceedings{djv1, title={Data-Juicer: A One-Stop Data Processing System for Large Language Models}, author={Daoyuan Chen and Yilun Huang and Zhijian Ma and Hesen Chen and Xuchen Pan and Ce Ge and Dawei Gao and Yuexiang Xie and Zhaoyang Liu and Jinyang Gao and Yaliang Li and Bolin Ding and Jingren Zhou}, booktitle={International Conference on Management of Data}, year={2024} }
@article{djv2, title={Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for Foundation Models}, author={Chen, Daoyuan and Huang, Yilun and Pan, Xuchen and Jiang, Nana and Wang, Haibin and Ge, Ce and Chen, Yushuo and Zhang, Wenhao and Ma, Zhijian and Zhang, Yilei and Huang, Jun and Lin, Wei and Li, Yaliang and Ding, Bolin and Zhou, Jingren}, journal={arXiv preprint arXiv:2501.14755}, year={2024} } ```
More data-related papers from the Data-Juicer Team:
> - (ICML'25 Spotlight) [Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development](https://arxiv.org/abs/2407.11784) - (CVPR'25) [ImgDiff: Contrastive Data Synthesis for Vision Large Language Models](https://arxiv.org/abs/2408.04594) - (TPAMI'25) [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://arxiv.org/abs/2407.08583) - (Benchmark Data) [HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data](https://arxiv.org/abs/2412.17574) - (Benchmark Data) [DetailMaster: Can Your Text-to-Image Model Handle Long Prompts?](https://www.arxiv.org/abs/2505.16915) - (Data Synthesis) [Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data](https://www.arxiv.org/abs/2502.04380) - (Data Synthesis) [MindGYM: What Matters in Question Synthesis for Thinking-Centric Fine-Tuning?](https://arxiv.org/abs/2503.09499) - (Data Scaling) [BiMix: A Bivariate Data Mixing Law for Language Model Pretraining](https://arxiv.org/abs/2405.14908)Owner
- Name: ModelScope
- Login: modelscope
- Kind: organization
- Email: contact@modelscope.cn
- Website: https://www.modelscope.cn/
- Repositories: 3
- Profile: https://github.com/modelscope
Model-as-a-Service in the making: bring accessible AI to all.
GitHub Events
Total
- Create event: 145
- Release event: 16
- Issues event: 163
- Watch event: 2,073
- Delete event: 173
- Member event: 6
- Issue comment event: 274
- Push event: 1,129
- Pull request review event: 477
- Pull request review comment event: 353
- Pull request event: 359
- Fork event: 93
Last Year
- Create event: 145
- Release event: 16
- Issues event: 163
- Watch event: 2,073
- Delete event: 173
- Member event: 6
- Issue comment event: 274
- Push event: 1,129
- Pull request review event: 477
- Pull request review comment event: 353
- Pull request event: 359
- Fork event: 93
Committers
Last synced: 6 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Yilun Huang | l****l@a****m | 118 |
| BeachWang | 1****7@p****n | 48 |
| Daoyuan Chen | 6****c | 45 |
| Ce Ge (ζη) | g****e@f****m | 35 |
| zhijianma | z****j@a****m | 30 |
| Cathy0908 | 3****8 | 20 |
| garyzhang99 | 4****9 | 16 |
| chenhesen | h****s@a****m | 12 |
| Xuchen Pan | 3****c | 11 |
| Cyrus Zhang | c****g@g****m | 11 |
| co63oc | c****c | 10 |
| Yuhan Liu | 3****x | 9 |
| cmgzn | 8****n | 8 |
| Zhen Qin | z****n@g****m | 7 |
| chenyushuo | 2****6@q****m | 5 |
| Qirui-jiao | 1****o | 3 |
| lingzhq | 1****q | 3 |
| 2108038773 | 1****3 | 2 |
| JamieYu | y****a@f****m | 2 |
| Yuexiang XIE | y****x@a****m | 2 |
| weijie | 3****o | 1 |
| simplaj | 3****j | 1 |
| seanzhang-zhichen | 7****n | 1 |
| ricksun2023 | 1****3 | 1 |
| panghu | 5****i | 1 |
| jackylee | q****1@g****m | 1 |
| Yanyi Liu | w****u@1****m | 1 |
| ShenQianli | s****i@u****u | 1 |
| Ruidong-X | x****g@g****m | 1 |
| NuODaniel | z****n@b****m | 1 |
| and 3 more... | ||
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 172
- Total pull requests: 593
- Average time to close issues: 30 days
- Average time to close pull requests: 9 days
- Total issue authors: 114
- Total pull request authors: 34
- Average comments per issue: 1.8
- Average comments per pull request: 0.42
- Merged pull requests: 433
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 113
- Pull requests: 420
- Average time to close issues: 26 days
- Average time to close pull requests: 6 days
- Issue authors: 83
- Pull request authors: 28
- Average comments per issue: 1.12
- Average comments per pull request: 0.34
- Merged pull requests: 303
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- BeachWang (10)
- yxdyc (10)
- drcege (9)
- HYLcool (6)
- abchbx (4)
- simplew2011 (4)
- javapythonphp (3)
- charonkk (3)
- HunterLG (3)
- DietDietDiet (2)
- wangpi26 (2)
- tian969 (2)
- baiyi-os (2)
- Yang-QW (2)
- xiafeng-nb (2)
Pull Request Authors
- HYLcool (121)
- BeachWang (82)
- drcege (62)
- yxdyc (48)
- Cathy0908 (38)
- liuyuhanalex (28)
- cyruszhang (27)
- garyzhang99 (26)
- cmgzn (23)
- co63oc (19)
- Qirui-jiao (17)
- pan-x-c (16)
- chenyushuo (12)
- zhenqincn (11)
- lingzhq (10)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 1,330 last-month
- Total dependent packages: 0
- Total dependent repositories: 0
- Total versions: 21
- Total maintainers: 1
pypi.org: py-data-juicer
Data Processing for and with Foundation Models.
- Documentation: https://py-data-juicer.readthedocs.io/
- License: Apache-2.0
-
Latest release: 1.4.2
published 6 months ago