cat-llm

CatLLM: A Reproducible LLM Pipeline for Classifying Open-Ended Text Across Domains

---

The Problem

If you work with open-ended text data survey responses, social media posts, academic papers, policy documents you know the pain: hundreds or thousands of free-text entries that need to be categorized before you can do any quantitative analysis. The traditional approach is manual coding either doing it yourself or hiring research assistants. It's slow, expensive, and doesn't scale.

The Solution

CatLLM is an ecosystem of Python packages that use LLMs to automate the categorization of open-ended text across domains. It handles:

• Category Assignment: Classify responses into your predefined categories (multi-label supported)
• Category Extraction: Automatically discover and extract categories from your data when you don't have a predefined scheme
• Category Exploration: Analyze category stability and saturation through repeated raw extraction
• Summarization: Generate concise summaries of text or PDF documents (paragraph, bullets, one-liner, structured, or full report)
• Prompt Optimization: Automatically tune classification prompts using user feedback for higher accuracy

With leading models like GPT-5, Gemini, and Qwen 3, CatLLM achieves 98% accuracy compared to human consensus on classification tasks.

Try the web app: https://huggingface.co/spaces/CatLLM/survey-classifier

---

Ecosystem

cat-llm is a meta-package (like tidyverse for R) that installs the full family of domain-specific classification packages. Each package can also be installed individually for a lighter footprint.

| Package | Domain | Install | Import | |---------|--------|---------|--------| | cat-llm | Everything (meta-package) | pip install cat-llm | import catllm | | cat-stack | General-purpose base | pip install cat-stack | import catstack | | cat-survey | Survey responses | pip install cat-survey | import catsurvey | | cat-vader | Social media | pip install cat-vader | import catvader | | cat-ademic | Academic papers | pip install cat-ademic | import catademic | | cat-pol | Political text | pip install cat-pol | import catpol | | cat-cog | Cognitive assessment | pip install cat-cog | import catcog | | cat-web | Web content | pip install cat-web | import catweb |

Dependency graph:

``` cat-stack general base + shared infra

cat-survey cat-vader cat-ademic domain packages (each depends on cat-stack) cat-pol cat-cog cat-web

         cat-llm                 meta-package (depends on all of the above)

```

Every domain package exposes the same five core functions classify(), extract(), explore(), summarize(), prompt_tune() with domain-specific parameters added on top. Learn once, apply anywhere.

---

Table of Contents

• Installation
  • R Package
• Quick Start
• Domain Packages
• Best Practices for Classification
• Configuration
• Supported Models
• API Reference
  • classify() - Unified function for text, image, and PDF (auto-detects input type)
  • prompt_tune() - Automatic prompt optimization via user feedback
  • extract() - Unified function for category extraction
  • explore() - Raw category extraction for saturation analysis
  • summarize() - Unified function for text and PDF summarization
  • imagescoredrawing()
  • image_features()
  • ceraddrawnscore()
• Deprecated Functions
• Related Projects
• Academic Research
• Testing
• Contributing & Support
• License

Installation

Install the full ecosystem: console pip install cat-llm

Or install only the domain you need (lighter footprint): console pip install cat-survey # survey responses pulls in cat-stack automatically pip install cat-vader # social media pip install cat-ademic # academic papers pip install cat-pol # political text pip install cat-cog # cognitive assessment (CERAD scoring) pip install cat-web # web content classification pip install cat-stack # general-purpose base only, no domain framing

Optional extras (apply to both cat-llm and cat-stack): console pip install cat-llm[pdf] # PDF support pip install cat-llm[embeddings] # Embedding-based similarity scores pip install cat-llm[formatter] # Local JSON formatter fallback

Dependencies

All dependencies are declared in pyproject.toml and installed automatically by pip. No manual dependency management is needed.

Core dependencies (installed with any pip install cat-llm or cat-stack):

| Package | Purpose | |---------|---------| | pandas | Data manipulation and output DataFrames | | tqdm | Progress bars during classification | | requests | HTTP calls to LLM provider APIs | | regex | JSON extraction from LLM responses |

Optional dependencies (install with extras syntax):

| Extra | Packages | Install | |-------|----------|---------| | pdf | PyMuPDF | pip install cat-llm[pdf] | | docx | python-docx | pip install cat-llm[docx] | | embeddings | sentence-transformers | pip install cat-llm[embeddings] | | formatter | torch, transformers, accelerate | pip install cat-llm[formatter] |

No provider-specific SDKs are required at runtime. CatLLM communicates with all LLM providers (OpenAI, Anthropic, Google, Mistral, HuggingFace, Perplexity, xAI, Ollama) via their REST APIs using requests directly.

R Package

An R interface is available for users who prefer R over Python. The R ecosystem mirrors the Python one with 8 packages (cat.stack, cat.survey, cat.vader, cat.ademic, cat.cog, cat.pol, cat.web, and the cat.llm meta-package) that wrap the Python code via reticulate.

```r

Install everything from R-universe (recommended)

install.packages("cat.llm", repos = c("https://chrissoria.r-universe.dev", "https://cloud.r-project.org"))

Install the Python backend (one-time setup)

library(cat.llm) installcatstack() ```

Or install a single domain package for a lighter footprint (e.g. install.packages("cat.survey", repos = ...)).

All core functions classify(), extract(), explore(), summarize(), plus domain-suffixed aliases like classify_survey() and classify_political() are available with native R syntax. See the R package README for the full ecosystem overview and per-package documentation.

---

Quick Start

This package is designed for building datasets at scale, not one-off queries. While you can categorize individual responses, its primary purpose is batch processing entire text columns, image collections, or PDF corpora into structured research datasets.

All outputs are formatted for immediate statistical analysis and can be exported directly to CSV.

Not to be confused with CAT-LLM for Chinese articlestyle transfer (Tao et al. 2024).

Option A via meta-package

Install cat-llm and access every domain through a single import:

```python import catllm

Domain-neutral classification (from cat-stack)

results = catllm.classify( inputdata=df['responses'], categories=["Positive", "Negative", "Neutral"], description="Customer feedback", apikey=api_key )

Survey classification adds survey-tuned prompts

results = catllm.classifysurvey( inputdata=df['responses'], categories=["Job change", "Family reasons", "Cost of living"], surveyquestion="Why did you move to a new city?", apikey=api_key )

Academic paper classification adds journal/field context

results = catllm.classifyacademic( inputdata=["paper1.pdf", "paper2.pdf"], categories=["Empirical", "Theoretical", "Review"], journalissn="0894-4393", apikey=api_key )

Social media classification adds platform context

results = catllm.classifysocial( inputdata=df['posts'], categories=["Misinformation", "Opinion", "News"], platform="Reddit", apikey=apikey )

Political text classification with built-in data sources

results = catllm.classifypolicy( source="citysandiego", categories=["Housing", "Public Safety", "Finance"], doctype="ordinance", since="2022-01-01", n=50, apikey=apikey )

Prompt optimization correct a small sample, get optimized prompts

result = catllm.prompttune( inputdata=df['responses'], categories=["Positive", "Negative", "Neutral"], apikey=apikey, sample_size=15, )

Cognitive assessment scoring

scores = catllm.ceraddrawnscore( shape="diamond", imageinput=df['drawingpaths'], apikey=apikey ) ```

Option B direct install (lighter footprint)

Install only the domain package you need:

```python

pip install cat-ademic

import catademic as cat

results = cat.classify( inputdata=["paper1.pdf", "paper2.pdf"], categories=["Empirical", "Theoretical", "Review"], journalissn="0894-4393", apikey=apikey ) ```

```python

pip install cat-vader

import catvader as cat

results = cat.classify( inputdata=df['posts'], categories=["Misinformation", "Opinion", "News"], platform="Reddit", apikey=api_key ) ```

```python

pip install cat-stack (general-purpose, no domain framing)

import catstack as cat

results = cat.classify( inputdata=df['textcolumn'], categories=["Category A", "Category B", "Category C"], description="My text data", apikey=apikey ) ```

---

Domain Packages

Each domain package wraps cat-stack's classification engine with domain-tuned prompts and domain-specific parameters. The base classify(), extract(), explore(), summarize(), and prompt_tune() parameters all work domain packages add parameters on top.

cat-survey Survey Responses

The survey package provides survey-tuned prompts, few-shot example support, and R/Stata wrappers. This was the original heart of cat-llm.

• Key parameter: survey_question= provides the survey question respondents were asked
• Supports few-shot examples (example1example6) for guiding classification
• R and Stata wrappers available for multi-language workflows

```python import catsurvey as cat

results = cat.classify( inputdata=df['responses'], categories=["Job change", "Family reasons", "Cost of living"], surveyquestion="Why did you move to a new city?", example1="I got a new job in Seattle|Job change", apikey=apikey ) ```

cat-vader Social Media

Platform-aware classification with social media metadata injection (platform, handle, hashtags, engagement metrics).

• Key parameter: platform= injects platform-specific context (Reddit, Twitter/X, forums)
• Handles nested comment structures and threaded conversations

```python import catvader as cat

results = cat.classify( inputdata=df['posts'], categories=["Misinformation", "Opinion", "News sharing"], platform="Reddit", apikey=api_key ) ```

cat-ademic Academic Papers

PDF-first classification for academic and long-form documents, with OpenAlex integration for fetching papers by journal, field, or topic.

• Key parameter: journal_issn= adds journal context for more accurate classification
• find_journal() helper for looking up journal metadata via OpenAlex
• Per-page and whole-document classification modes

```python import catademic as cat

results = cat.classify( inputdata=["paper1.pdf", "paper2.pdf"], categories=["Empirical", "Theoretical", "Review"], journalissn="0894-4393", apikey=apikey ) ```

cat-pol Political Text

Domain-tuned prompts for political science legislation, speeches, executive orders, social media. Includes 15 built-in political data sources on HuggingFace (municipal ordinances from 11 California cities, federal public laws, executive orders, presidential speeches, and Trump's Truth Social posts).

• Key parameter: source= pull data directly from built-in political datasets
• Key parameter: document_context= frames the political text type for better classification
• Built-in sources updated weekly via automated scrapers
• Designed for policy area coding, ideology classification, cross-city comparison

```python import catpol as pol

Classify ordinances from a built-in source

results = pol.classify( source="citysandiego", categories=["Housing", "Public Safety", "Infrastructure", "Finance"], doctype="ordinance", since="2022-01-01", n=50, apikey=api_key, )

Optimize prompts with user feedback

result = pol.prompttune( source="citysandiego", categories=["Pro-Business", "Pro-Regulation", "Tax Increase"], apikey=apikey, samplesize=15, )

Summarize executive orders as bullet points

summaries = pol.summarize( source="federalexecutiveorders", n=10, format="bullets", apikey=apikey, )

Fetch raw data (no API key needed)

df = pol.fetchsource("socialtrumptruth", since="2024-01-01") pol.listsources() # see all 15 sources ```

cat-cog Cognitive Assessment

LLM-powered evaluation of drawn images for neuropsychological testing, including CERAD scoring.

• Key function: cerad_drawn_score() scores drawings of circles, diamonds, rectangles, and cubes
• Designed for clinical research and cognitive screening studies

```python import catcog

scores = catcog.ceraddrawnscore( shape="diamond", imageinput=df['drawingpaths'], apikey=apikey ) ```

cat-web Web Content

Web content classification and fact-checking. Thin wrapper on cat-stack for URL-based classification.

```python import catweb as cat

results = cat.classify( inputdata=df['urls'], categories=["News", "Blog", "E-commerce", "Academic"], apikey=api_key ) ```

cat-stack General-Purpose Base

The domain-neutral classification engine that all other packages build on. Use this directly when your text doesn't fit neatly into a specific domain.

```python import catstack as cat

results = cat.classify( inputdata=df['textcolumn'], categories=["Category A", "Category B", "Category C"], description="My text data", apikey=apikey ) ```

---

Best Practices for Classification

These recommendations are based on empirical testing across 4 surveys, 4 models (7B to frontier-class), and 250-row subsamples compared against human-coded ground truth.

What works

• Detailed category descriptions: The single biggest lever for accuracy. Instead of short labels like "Job change", use verbose descriptions like "The person had a job or school or career change, including transferred and retired." This consistently improves accuracy across all models by several percentage points.
• Include an "Other" category: Adding a catch-all category like "Other: The response does not fit any of the above categories." prevents the model from forcing ambiguous responses into ill-fitting categories, improving precision.
• Few-shot examples (example1example6): Providing 24 labeled examples can help, particularly for weaker models. Effects are modest (+01 pp on average) and model-dependent.
• Low temperature (creativity=0): For classification tasks, deterministic output is generally preferable. Higher temperatures introduce noise without improving accuracy.

What doesn't help (or hurts)

• Chain of Thought (chain_of_thought): In our testing, enabling COT did not improve classification accuracy for any model and slightly degraded it for some. It is now off by default.
• Chain of Verification (chain_of_verification): CoVE uses ~4x the API calls per response for a self-verification loop. Despite the added cost, it consistently reduced accuracy by 12 percentage points, primarily by retracting correct classifications during the verification step. Not recommended for classification tasks.
• Step-back prompting (step_back_prompt): Results were inconsistent slight gains for weaker models (~+1.8 pp) but slight losses for stronger models (~-0.5 pp), with high variance across surveys. Not recommended as a default strategy.
• Context prompting (context_prompt): Adds generic expert context to the prompt. No consistent benefit observed.

Summary

The most effective approach is straightforward: write detailed category descriptions, include an "Other" category, and use a capable model at low temperature. Advanced prompting strategies add complexity and cost without reliable gains for classification tasks.

---

Configuration

Get Your API Key

Get an API key from your preferred provider:

• OpenAI: platform.openai.com
• Anthropic: console.anthropic.com
• Google: aistudio.google.com
• Huggingface: huggingface.co/settings/tokens
• xAI: console.x.ai
• Mistral: console.mistral.ai
• Perplexity: perplexity.ai/settings/api

Most providers require adding a payment method and purchasing credits. Store your key securely and never share it publicly.

Supported Models

• OpenAI: GPT-4o, GPT-4, GPT-5, etc.
• Anthropic: Claude Sonnet 4, Claude 3.5 Sonnet, Claude Haiku, etc.
• Google: Gemini 2.5 Flash, Gemini 2.5 Pro, etc.
• Huggingface: Qwen, Llama 4, DeepSeek, and thousands of community models
• xAI: Grok models
• Mistral: Mistral Large, Pixtral, etc.
• Perplexity: Sonar Large, Sonar Small, etc.

Fully Tested: - OpenAI (GPT-4, GPT-4o, GPT-5, etc.) - Anthropic (Claude Sonnet 4, Claude 3.5 Sonnet, Haiku) - Perplexity (Sonar models) - Google Gemini - Free tier has severe rate limits (5 RPM). Requires Google AI Studio billing account for large-scale use. - Huggingface - Access to Qwen, Llama 4, DeepSeek, and thousands of user-trained models for specific tasks. API routing can occasionally be unstable. - xAI (Grok models) - Mistral (Mistral Large, Pixtral, etc.)

Note: For best results, I recommend starting with OpenAI or Anthropic.

API Reference

Note: The functions documented below are the domain-neutral versions from cat-stack. They work with any text, image, or PDF data without domain-specific framing. Domain packages (cat-survey, cat-vader, cat-ademic, etc.) accept all the same parameters and add domain-specific ones on top (e.g., survey_question=, platform=, journal_issn=). See Domain Packages for details.

classify()

Unified classification function for text, image, and PDF inputs. Input type is auto-detected from your datano need to specify whether you're classifying text, images, or PDFs.

Supports both single-model and multi-model ensemble classification for improved accuracy through consensus voting.

Parameters: - input_data: The data to classify. Can be: - Text: list of strings or pandas Series - Images: directory path, single file, or list of image paths - PDFs: directory path, single file, or list of PDF paths - categories (list): List of category names for classification - api_key (str): API key for the LLM service (single-model mode) - description (str): Description of the input data context - user_model (str, default="gpt-4o"): Model to use - mode (str, default="image"): PDF processing mode - "image", "text", or "both" - creativity (float, optional): Temperature setting (0.0-1.0) - survey_question (str, default=""): The survey question respondents were asked. Provides important context for classification. - safety (bool, default=False): Save progress after each row. If the process fails midway, you won't lose your work. Requires filename. - chain_of_thought (bool, default=False): Enable step-by-step reasoning within a single prompt. Low cost increase. - context_prompt (bool, default=False): Add expert analyst role to the prompt. Minimal cost increase. - step_back_prompt (bool, default=False): Ask the model to consider broader conceptual background before classifying. Moderate cost increase. - chain_of_verification (bool, default=False): Multi-prompt verification loop where the model checks its own work. High cost increase (3-5x). - example1example6 (str, optional): Few-shot examples to guide classification (up to 6). - filename (str, optional): Output filename for CSV - save_directory (str, optional): Directory to save results - model_source (str, default="auto"): Provider - "auto", "openai", "anthropic", "google", "mistral", "perplexity", "huggingface", "xai" - multi_label (bool, default=True): If True, multiple categories can be assigned per input (multi-label). If False, the model picks the single best category (single-label). Output format is unchangedstill one 0/1 column per category. - categories_per_call (int, default=None): Maximum number of categories per LLM call. When set, splits the category list into chunks, runs a separate call per chunk, and merges results. Each chunk automatically gets a temporary "Other" catch-all to improve accuracy. A unified "Other" column is added to the output when all real categories are 0 but at least one chunk's "Other" fired. Useful for large category sets (20+). Not supported with batch_mode=True. - models (list, optional): For multi-model ensemble, list of (model, provider, api_key) or (model, provider, api_key, config_dict) tuples - consensus_threshold (str or float, default="unanimous"): Agreement threshold for ensemble mode. Options: "unanimous" (100%, default best accuracy), "majority" (50%), "two-thirds" (67%), or a custom float between 0 and 1. - parallel (bool, default=None): Controls concurrent vs sequential model execution in ensemble mode. None (default) auto-detects: sequential for local models (Ollama), parallel for cloud providers. Set True to force parallel or False to force sequential. Sequential mode is useful for resource-constrained environments or debugging. - batch_mode (bool, default=False): (Experimental) Submit the entire job as an async batch request instead of making synchronous calls. Supported providers: OpenAI, Anthropic, Google, Mistral, xAI. Reduces API costs by ~50%. Works with both single-model and multi-model ensemble (each model submits its own batch job concurrently; providers without batch API fall back to synchronous calls). Not compatible with PDF/image inputs. The function blocks until the batch completes. - batch_poll_interval (float, default=30.0): Seconds between status polls when batch_mode=True. - batch_timeout (float, default=86400.0): Maximum seconds to wait for a batch job before raising BatchJobExpiredError. - embeddings (bool, default=False): Add embedding-based similarity scores alongside binary 0/1 classifications. Adds category_N_similarity columns (01 float) using a local sentence-transformer model (BAAI/bge-small-en-v1.5, ~130MB). Text input only (skipped for PDF/image). Requires pip install cat-llm[embeddings]. - category_descriptions (dict, optional): Richer text descriptions per category for embedding similarity (e.g., {"Past_Support": "References to help received from family"}). Only used when embeddings=True. - embedding_tiebreaker (bool, default=False): When ensemble consensus produces a tie (equal votes for 0 and 1), use embedding centroid similarity to break the tie. Requires pip install cat-llm[embeddings]. - min_centroid_size (int, default=3): Minimum number of confirmed-positive responses needed to build a reliable centroid for embedding_tiebreaker. If fewer positives exist, falls back to raw similarity against the category text. - json_formatter (bool, default=False): Use a local fine-tuned model to fix malformed JSON output before marking responses as failed. The formatter runs only when extract_json() produces invalid outputzero cost on the happy path. On first use, the model (~1GB) is downloaded from HuggingFace Hub. Requires pip install cat-llm[formatter]. - add_other (str or bool, default="prompt"): Controls auto-addition of an "Other" catch-all category. "prompt" asks the user, True adds silently, False never adds. - check_verbosity (bool, default=True): Check whether categories have descriptions and examples (1 API call). Set to False to skip. - use_json_schema (bool, default=True): Use structured JSON schema for LLM output. Set to False for providers that don't support it well. - max_categories (int, default=12): Maximum categories for auto-extraction when categories="auto". - categories_per_chunk (int, default=10): Categories to extract per chunk during auto-extraction. - divisions (int, default=10): Number of chunks to divide data into during auto-extraction. - research_question (str, optional): Research context to guide classification. - row_delay (float, default=0.0): Seconds to wait between processing each row. Useful for rate-limited APIs (e.g., Google free tier at 5 RPM). - max_retries (int, default=5): Maximum number of retries for failed API calls per row. - retry_delay (float, default=1.0): Base delay in seconds between retries (uses exponential backoff). - fail_strategy (str, default="partial"): How to handle rows that fail after all retries. "partial" returns results with failed rows marked; "strict" raises an error on any failure. - max_workers (int, default=None): Maximum parallel workers for API calls. None auto-selects. - auto_download (bool, default=False): Automatically download missing Ollama models without prompting. - progress_callback (callable, optional): Callback function for progress updates. Called as progress_callback(current_step, total_steps). - pdf_dpi (int, default=150): DPI resolution for rendering PDF pages as images. Higher values improve quality but increase processing time and cost. - thinking_budget (int, default=0): Token budget for model reasoning/thinking. Set to 0 to disable. Behavior varies by provider:

| Provider | thinking_budget=0 | thinking_budget > 0 (e.g., 8192) | |----------|---------------------|--------------------------------------| | OpenAI | reasoning_effort="minimal" | reasoning_effort="high" | | Anthropic | Thinking disabled | Extended thinking enabled (min 1024 tokens, forces temperature=1) | | Google | Thinking disabled | thinkingConfig: {thinkingBudget: N} (min 128 tokens) | | HuggingFace | Thinking disabled (or use non-thinking model variant) | Thinking enabled (or use thinking model variant) |

Note: Mistral and xAI models do not have reasoning/thinking toggles thinking_budget has no effect on these providers. For Qwen3 on HuggingFace, reasoning is controlled by choosing the model variant: use Qwen3-VL-235B-A22B-Thinking for reasoning or Qwen3-VL-235B-A22B-Instruct for standard mode with thinking_budget=0.

Returns: - pandas.DataFrame: Classification results with category columns

Examples:

```python import catllm as cat

Text classification (auto-detected)

results = cat.classify( inputdata=df['responses'], categories=["Positive feedback", "Negative feedback", "Neutral"], description="Customer satisfaction survey", apikey=api_key )

Image classification (auto-detected from file paths)

results = cat.classify( inputdata="/path/to/images/", categories=["Contains person", "Outdoor scene", "Has text"], description="Product photos", apikey=api_key )

PDF classification (auto-detected, processes each page separately)

results = cat.classify( inputdata="/path/to/reports/", categories=["Contains table", "Has chart", "Is summary page"], description="Financial reports", mode="both", # Use both image and extracted text apikey=api_key )

Single-label classification (pick one best category per response)

results = cat.classify( inputdata=df['responses'], categories=["Positive", "Negative", "Neutral"], multilabel=False, apikey=apikey )

Multi-model ensemble for higher accuracy

results = cat.classify( inputdata=df['responses'], categories=["Positive", "Negative", "Neutral"], models=[ ("gpt-4o", "openai", "sk-..."), ("claude-sonnet-4-20250514", "anthropic", "sk-ant-..."), ("gemini-2.5-flash", "google", "AIza..."), ], consensusthreshold="unanimous", ) ```

Multi-Model Ensemble:

When you provide the models parameter, CatLLM runs classification across multiple models and combines results using majority voting. This can significantly improve accuracy by reducing individual model biases. By default, cloud models run in parallel while local models (Ollama) run sequentially controlled by the parallel parameter.

The output includes: - Individual model predictions (e.g., category_1_gpt_4o, category_1_claude) - Consensus columns (e.g., category_1_consensus) - Agreement scores showing how many models agreed

---

extract()

Unified category extraction function for text, image, and PDF inputs. Automatically discovers categories in your data when you don't have a predefined scheme.

Planned improvement: Allow specifying a separate, more powerful model for the semantic merge step (e.g., use GPT-4o-mini for bulk extraction, GPT-4o for the final consolidation). This "tiered" approach could improve merge quality without significantly increasing cost.

Parameters: - input_data: The data to explore (text list, image paths, or PDF paths) - api_key (str): API key for the LLM service - input_type (str, default="text"): Type of input - "text", "image", or "pdf" - survey_question (str, default=""): The survey question or description of the data. Provides context for category discovery. - description (str, optional): Deprecated alias for survey_question. Use survey_question instead. - max_categories (int, default=12): Maximum number of final categories to return - categories_per_chunk (int, default=10): Categories to extract per chunk - divisions (int, default=12): Number of chunks to divide data into - iterations (int, default=8): Number of extraction passes over the data - user_model (str, default="gpt-4o"): Model to use - model_source (str, default="auto"): Provider - "auto", "openai", "anthropic", "google", etc. - creativity (float, optional): Temperature setting (0.0-1.0). None uses model default. - specificity (str, default="broad"): "broad" or "specific" category granularity - research_question (str, optional): Research context to guide extraction - focus (str, optional): Focus instruction for category extraction (e.g., "emotional responses") - mode (str, default="text"): Processing mode for non-text inputs - "text", "image", or "both" - filename (str, optional): Output filename for CSV - random_state (int, optional): Random seed for reproducibility of chunk sampling - chunk_delay (float, default=0.0): Seconds to wait between processing each chunk. Useful for rate-limited APIs. - auto_download (bool, default=False): Automatically download missing Ollama models without prompting. - progress_callback (callable, optional): Callback function for progress updates.

Default parameter rationale: The defaults of divisions=12 and iterations=8 were determined through empirical analysis. We ran a 6x6 grid search over [1, 4, 8, 12, 16, 20] for both parameters, repeating each combination 10 times and measuring pairwise Jaro-Winkler consistency across runs. Consistency peaked at 12 divisions and 8 iterations, with values beyond this point offering no meaningful improvement.

Returns: - dict with keys: - counts_df: DataFrame of categories with counts - top_categories: List of top category names - raw_top_text: Raw model output

Example:

```python import catllm as cat

Extract categories from survey responses

results = cat.extract( inputdata=df['responses'], surveyquestion="Why did you move?", apikey=apikey, max_categories=10, focus="decisions to relocate" # Optional focus )

print(results['top_categories'])

['Employment opportunity', 'Family reasons', 'Cost of living', ...]

```

---

explore()

Raw category extraction for frequency and saturation analysis. Unlike extract(), which normalizes, deduplicates, and semantically merges categories into a clean final set, explore() returns every category string from every chunk across every iteration with duplicates intact.

This is useful for analyzing which categories are robust (consistently discovered across runs) versus which are noise (appearing only once or twice). By increasing iterations, you can build saturation curves showing when category discovery converges.

Parameters: - input_data: List of text responses or pandas Series - api_key (str): API key for the LLM service - description (str): The survey question or description of the data - max_categories (int, default=12): Maximum categories passed through to the extraction prompt. - categories_per_chunk (int, default=10): Categories to extract per chunk - divisions (int, default=12): Number of chunks to divide data into - user_model (str, default="gpt-4o"): Model to use - model_source (str, default="auto"): Provider - "auto", "openai", "anthropic", "google", etc. - creativity (float, optional): Temperature setting (0.0-1.0). None uses model default. - specificity (str, default="broad"): "broad" or "specific" category granularity - research_question (str, optional): Research context to guide extraction - focus (str, optional): Focus instruction (e.g., "decisions to relocate") - iterations (int, default=8): Number of passes over the data - random_state (int, optional): Random seed for reproducibility - filename (str, optional): Output CSV filename (one category per row) - chunk_delay (float, default=0.0): Seconds to wait between processing each chunk. Useful for rate-limited APIs. - auto_download (bool, default=False): Automatically download missing Ollama models without prompting. - progress_callback (callable, optional): Callback function for progress updates.

Returns: - list[str]: Every category extracted from every chunk across every iteration. Length iterations divisions categories_per_chunk.

Example:

```python import catllm as cat

Run extraction with many iterations for saturation analysis

rawcategories = cat.explore( inputdata=df['responses'], description="Why did you move?", apikey=apikey, iterations=20, divisions=5, categoriesperchunk=10, )

Count how often each category appears across runs

from collections import Counter counts = Counter(rawcategories) for category, freq in counts.mostcommon(15): print(f"{freq:3d}x {category}") ```

---

summarize()

Unified summarization function for text and PDF inputs. Generates concise summaries of survey responses, documents, or any text data. Input type is auto-detected from your data.

Supports both single-model and multi-model ensemble summarization. In multi-model mode, summaries from all models are synthesized into a consensus summary.

Parameters: - input_data: The data to summarize. Can be: - Text: list of strings, pandas Series, or single string - PDF: directory path, single PDF path, or list of PDF paths - api_key (str): API key for the LLM service (single-model mode) - description (str): Description of what the content contains (provides context) - instructions (str): Specific summarization instructions (e.g., "bullet points") - max_length (int): Maximum summary length in words - focus (str): What to focus on (e.g., "main arguments", "emotional content") - user_model (str, default="gpt-4o"): Model to use - model_source (str, default="auto"): Provider - "auto", "openai", "anthropic", "google", etc. - creativity (float, optional): Temperature setting (0.0-1.0). None uses model default. - thinking_budget (int, default=0): Token budget for extended thinking/reasoning. See classify() for provider-specific behavior. - chain_of_thought (bool, default=True): Enable step-by-step reasoning. On by default for summarization. - context_prompt (bool, default=False): Add expert analyst role to the prompt. - step_back_prompt (bool, default=False): Ask the model to consider broader context before summarizing. - mode (str, default="image"): PDF processing mode: - "image": Render pages as images (best for visual documents) - "text": Extract text only (faster, good for text-heavy PDFs) - "both": Send both image and extracted text (most comprehensive) - filename (str): Output CSV filename - save_directory (str): Directory to save results - pdf_dpi (int, default=150): DPI resolution for rendering PDF pages as images. - models (list): For multi-model mode, list of (model, provider, api_key) tuples - parallel (bool, default=None): Controls concurrent vs sequential model execution. None auto-detects (sequential for Ollama, parallel for cloud). - max_workers (int, default=None): Maximum parallel workers for API calls. - auto_download (bool, default=False): Automatically download missing Ollama models without prompting. - progress_callback (callable, optional): Callback function for progress updates. - safety (bool, default=False): If True, saves progress to CSV after each item. Requires filename. - max_retries (int, default=5): Max retries per API call. - batch_retries (int, default=2): Number of batch retry passes for failed items. - retry_delay (float, default=1.0): Delay between retries in seconds. - row_delay (float, default=0.0): Delay in seconds between processing each row. Useful to avoid rate limits. - fail_strategy (str, default="partial"): How to handle failures "partial" keeps successful results, "strict" blanks the row if any model fails. - batch_mode (bool, default=False): If True, use async batch API (50% cost savings). Supported providers: openai, anthropic, google, mistral, xai. Not compatible with PDF input. - batch_poll_interval (float, default=30): Seconds between batch job status checks. - batch_timeout (float, default=86400): Max seconds to wait for batch completion (default 24h).

Returns: - pandas.DataFrame: Results with summary columns: - survey_input: Original text or page label (for PDFs) - summary: Generated summary (or consensus for multi-model) - processing_status: "success", "error", "skipped" - pdf_path: Path to source PDF (PDF mode only) - page_index: Page number, 0-indexed (PDF mode only)

Examples:

```python import catllm as cat

Single model text summarization

results = cat.summarize( inputdata=df['responses'], description="Customer feedback", apikey=api_key )

PDF summarization (auto-detected from file paths)

results = cat.summarize( inputdata="/path/to/pdfs/", description="Research papers", mode="image", apikey=api_key )

PDF summarization with specific files and focus

results = cat.summarize( inputdata=["doc1.pdf", "doc2.pdf"], description="Financial reports", mode="both", focus="key metrics and trends", maxlength=100, apikey=apikey )

With safety saves and row delay

results = cat.summarize( inputdata=df['responses'], description="Customer feedback", apikey=apikey, safety=True, filename="results.csv", rowdelay=1.0, )

Batch mode (50% cost savings)

results = cat.summarize( inputdata=df['responses'], description="Customer feedback", apikey=apikey, batchmode=True, filename="batch_results.csv", )

Multi-model with synthesis

results = cat.summarize( input_data=df['responses'], models=[ ("gpt-4o", "openai", "sk-..."), ("claude-sonnet-4-20250514", "anthropic", "sk-ant-..."), ], ) ```

---

image_score_drawing()

Performs quality scoring of images against a reference description and optional reference image, returning structured results with optional CSV export.

Methodology: Processes each image individually, assigning a drawing quality score on a 5-point scale based on similarity to the expected description:

• 1: No meaningful similarity (fundamentally different)
• 2: Barely recognizable similarity (25% match)
• 3: Partial match (50% key features)
• 4: Strong alignment (75% features)
• 5: Near-perfect match (90%+ similarity)

Parameters: - reference_image_description (str): A description of what the model should expect to see - image_input (list): List of image file paths or folder path containing images - reference_image (str): A file path to the reference image - api_key (str): API key for the LLM service - user_model (str, default="gpt-4o"): Specific vision model to use - creativity (float, default=0): Temperature/randomness setting (0.0-1.0) - safety (bool, default=False): Enable safety checks and save results at each API call step - filename (str, default="imagescores.csv"): Filename for CSV output - `savedirectory(str, optional): Directory path to save the CSV file -model_source` (str, default="OpenAI"): Model provider

Returns: - pandas.DataFrame: DataFrame with image paths, quality scores, and analysis details

Example:

```python import catllm as cat

imagescores = cat.imagescoredrawing( referenceimagedescription='A hand-drawn circle', imageinput=['image1.jpg', 'image2.jpg', 'image3.jpg'], usermodel="gpt-4o", apikey="OPENAIAPIKEY" ) ```

---

image_features()

Extracts specific features and attributes from images, returning exact answers to user-defined questions (e.g., counts, colors, presence of objects).

Methodology: Processes each image individually using vision models to extract precise information about specified features. Unlike scoring and classification functions, this returns factual data such as object counts, color identification, or presence/absence of specific elements.

Parameters: - image_description (str): A description of what the model should expect to see - image_input (list): List of image file paths or folder path containing images - features_to_extract (list): List of specific features to extract (e.g., ["number of people", "primary color", "contains text"]) - api_key (str): API key for the LLM service - user_model (str, default="gpt-4o"): Specific vision model to use - creativity (float, default=0): Temperature/randomness setting (0.0-1.0) - to_csv (bool, default=False): Whether to save the output to a CSV file - safety (bool, default=False): Enable safety checks and save results at each API call step - filename (str, default="categorizeddata.csv"): Filename for CSV output - `savedirectory(str, optional): Directory path to save the CSV file -model_source` (str, default="OpenAI"): Model provider

Returns: - pandas.DataFrame: DataFrame with image paths and extracted feature values

Example:

```python import catllm as cat

features = cat.imagefeatures( imagedescription='Product photos from e-commerce site', featurestoextract=['number of items', 'primary color', 'has price tag'], imageinput='/path/to/images/', usermodel="gpt-4o", apikey="OPENAIAPI_KEY" ) ```

---

cerad_drawn_score()

Automatically scores drawings of circles, diamonds, overlapping rectangles, and cubes according to the official Consortium to Establish a Registry for Alzheimer's Disease (CERAD) scoring system.

Methodology: Processes each image individually, evaluating the drawn shapes based on CERAD criteria. Works even with images that contain other drawings or writing.

Parameters: - shape (str): The type of shape to score ("circle", "diamond", "rectangles", "cube") - image_input (list): List of image file paths or folder path containing images - api_key (str): API key for the LLM service - user_model (str, default="gpt-4o"): Specific model to use - creativity (float, default=0): Temperature/randomness setting (0.0-1.0) - safety (bool, default=False): Enable safety checks and save results at each API call step - filename (str, optional): Filename for CSV output - model_source (str, default="auto"): Model provider

Returns: - pandas.DataFrame: DataFrame with image paths, CERAD scores, and analysis details

Example:

```python import catllm as cat

diamondscores = cat.ceraddrawnscore( shape="diamond", imageinput=df['diamondpicpath'], apikey=apikey, safety=True, filename="diamond_scores.csv", ) ```

---

Deprecated Functions

The following functions are deprecated and will be removed in a future version. Please use classify() instead, which auto-detects input type and supports all the same features.

| Deprecated Function | Replacement | |---------------------|-------------| | multi_class() | classify(input_data=texts, ...) | | image_multi_class() | classify(input_data=images, ...) | | pdf_multi_class() | classify(input_data=pdfs, ...) | | explore_corpus() | extract(input_data=texts, ...) | | explore_common_categories() | extract(input_data=texts, ...) |

These functions still work but will show deprecation warnings. Migration is straightforwardsimply use classify() with your data and it will automatically detect whether you're passing text, images, or PDFs.

---

Related Projects

Looking for web research capabilities? Check out llm-web-research - a precision-focused LLM-powered web research tool that uses a novel Funnel of Verification (FoVe) methodology to reduce false positives. It's designed for use cases where accuracy matters more than completeness.

bash pip install llm-web-research

Academic Research

This package implements methodology from research on LLM performance in social science applications, including the UC Berkeley Social Networks Study. The package addresses reproducibility challenges in LLM-assisted research by providing standardized interfaces and consistent output formatting.

If you use this package for research, please cite:

Soria, C. (2025). CatLLM (0.1.0). Zenodo. https://doi.org/10.5281/zenodo.15532317

Testing

To verify that CatLLM is working correctly, you can run a local classification test using Ollama (free, no API keys required):

1. Install Ollama and pull a small model: console ollama pull llama3.1:8b

2. Run the local classification example notebook: examples/Classifying Text with Local Models (Ollama).ipynb

This notebook walks through text classification using a local model, verifying that the full pipeline prompt construction, structured JSON output, and result parsing works end-to-end without any cloud API calls.

For cloud provider testing, additional example notebooks are available in the examples/ directory covering ensemble classification, category extraction, and summarization.

Contributing & Support

Contributions are welcome! Please see CONTRIBUTING.md for detailed guidelines.

• Report bugs or request features: Open a GitHub Issue
• Ask questions or get help: GitHub Discussions or Issues
• Contribute code: Fork the repo, create a branch, and submit a pull request see CONTRIBUTING.md
• Research collaboration: Email ChrisSoria@Berkeley.edu

License

cat-llm is distributed under the terms of the GNU license.