Accelerate Data Processing and Streamline Synthetic Data Generation with NVIDIA NeMo Curator

NeMo Curator, part of the NVIDIA NeMo software suite for managing the AI agent lifecycle, is a Python library specifically designed for fast and scalable data processing and curation for generative AI use cases such as foundation language model pretraining, text-to-image model training, domain-adaptive pretraining (DAPT), supervised fine-tuning (SFT) and parameter-efficient fine-tuning (PEFT).

It greatly accelerates data processing and curation by leveraging GPUs with Dask and RAPIDS, resulting in significant time savings. The library provides a customizable and modular interface, simplifying pipeline expansion and accelerating model convergence through the preparation of high-quality tokens.

NeMo Curator also provides pre-built pipelines for synthetic data generation for customization and evaluation of generative AI systems. You can use any OpenAI API compatible model and plug it in NeMo Curator's synthetic data generation pipelines to process and curate high-quality synthetic data for various use cases.

Getting Started

New to NeMo Curator? Start with our quickstart guides for hands-on experience:

• Text Curation Quickstart - Set up your environment and run your first text curation pipeline in under 30 minutes
• Image Curation Quickstart - Learn to curate large-scale image-text datasets for generative model training

For production deployments and advanced configurations, see our Setup & Deployment documentation.

Key Features

With NeMo Curator, you can process raw data and curate high-quality data for training and customizing generative AI models such as LLMs, VLMs and WFMs. NeMo Curator provides a collection of scalable data processing modules for text and image curation.

Text Curation

All of our text pipelines have great multilingual support. With NeMo Curator, you can pick and choose the features you want and build your data curation pipelines. Text curation follows a three-stage workflow: Load → Process → Generate. A typical pipeline starts by downloading raw data from public resources, then applies cleaning and filtering steps, and optionally generates synthetic data for training enhancement.

Load Data

• Download and Extraction - Default implementations for Common Crawl, Wikipedia, and ArXiv sources with easy customization for other sources

Process Data

• Quality Assessment & Filtering

  • Heuristic Filtering - 30+ heuristic filters for punctuation density, length, and repetition analysis
  • fastText Classification - Fast language and quality classification
  • GPU-Accelerated Classification - Domain, Quality, Safety, Educational Content, Content Type, and Prompt Task/Complexity Classification

• Deduplication

  • Exact Deduplication - Remove identical documents efficiently
  • Fuzzy Deduplication - MinHash Locality Sensitive Hashing with optional False Positive Check
  • Semantic Deduplication - GPU-accelerated semantic deduplication using RAPIDS cuML, cuDF, and PyTorch

• Content Processing & Cleaning

  • Text Cleaning - Remove improperly decoded Unicode characters, inconsistent line spacing, and excessive URLs
  • PII Redaction - Identify and remove personally identifiable information from training datasets

• Specialized Processing

  • Language Identification - Accurate language detection using fastText
  • Task Decontamination - Remove potential evaluation data leakage from training datasets

Generate Data

• Synthetic Data Pipelines - Pre-built pipelines for generating high-quality synthetic training data:
  • Open Q&A Generation - Create question-answer pairs for instruction tuning
  • Math Problem Generation - Generate mathematical problems for educational content
  • Coding Tasks - Create programming challenges and code examples
  • Writing Prompts - Generate creative writing and content creation tasks
  • Dialogue Generation - Create conversational data for chat models
  • Nemotron Pipelines - Wikipedia-style rewriting and knowledge distillation

Image Curation

NeMo Curator provides powerful image curation features to curate high-quality image data for training generative AI models such as LLMs, VLMs, and WFMs. Image curation follows a Load → Process workflow: download datasets in WebDataset format, create embeddings, apply quality filters (NSFW and Aesthetic), and remove duplicates using semantic deduplication.

Load Data

• WebDataset Loading - Load large-scale image-text datasets in WebDataset format

Process Data

• Embeddings & Feature Extraction

  • Image Embedding Creation - Generate CLIP embeddings for image analysis

• Quality Assessment & Filtering

  • Aesthetic Classification - Filter images based on aesthetic quality
  • NSFW Classification - Remove inappropriate content from datasets

• Deduplication

  • Semantic Deduplication - Remove visually similar images using embedding-based clustering

Module Ablation and Compute Performance

The modules within NeMo Curator were primarily designed to process and curate high-quality documents at scale. To evaluate the quality of the data, we curated Common Crawl documents and conducted a series of ablation experiments. In these experiments, we trained a 357M-parameter GPT-style model using datasets generated at various stages of our data curation pipeline, which was implemented in NeMo Curator.

The following figure shows that the use of different data curation modules implemented in NeMo Curator led to improved model zero-shot downstream task performance.

NeMo Curator leverages NVIDIA RAPIDS™ libraries like cuDF, cuML, and cuGraph along with Dask to scale workloads across multi-node, multi-GPU environments, significantly reducing data processing time. With NeMo Curator, developers achieve approximately 16× faster fuzzy‑deduplication on an 8 TB RedPajama‑v2 subset, with ~40% lower TCO and near‑linear scaling on 1–4 H100 80 GB nodes. Refer to the chart below to learn more details.

NeMo Curator exhibits near‑linear scaling for fuzzy deduplication. On an 8 TB RedPajama‑v2 subset (~1.78 trillion tokens), processing time drops from 2.05 hours on one H100 80 GB node to 0.50 hours on four nodes. Refer to the scaling chart below to learn more:

Contribute to NeMo Curator

We welcome community contributions! Please refer to CONTRIBUTING.md for the process.