Critic Rubrics

Type-safe function-calling-based LLM-as-judge evaluation framework for structured prediction and analysis.

[!WARNING] This repository is an active research project. APIs and implementations are subject to major changes. Use with caution.

To Install

bash pip install git+https://github.com/All-Hands-AI/critic-rubrics

Core Data Structures

Prediction Types

All predictions inherit from BasePrediction and define how data is flattened into OpenAI tool schemas:

```python from critic_rubrics import BinaryPrediction, TextPrediction, ClassificationPrediction from typing import Literal

Boolean detection with evidence

class BinaryPrediction(BasePrediction): detected: bool # Flattened as: detected rationale: str # Flattened as: rationale

Free text output

class TextPrediction(BasePrediction): text: str # Flattened as: _text

Single-label classification with evidence

class ClassificationPredictionL: label: L # Flattened as: (with enum constraint) rationale: str # Flattened as: _rationale ```

Feature Definition

```python from critic_rubrics import Feature

feature = Feature( name="taskcomplexity", description="Assess the complexity level of the given task", predictiontype=ClassificationPrediction[Literal["simple", "moderate", "complex"]] ) ```

Feature Data

```python from critic_rubrics import FeatureData

Combines feature definition with actual prediction data

feature_data = FeatureData( feature=feature, prediction=ClassificationPrediction(label="complex", rationale="Multiple dependencies") ) ```

Core APIs

Rubric Definition

```python from critic_rubrics import BaseRubrics, Feature, BinaryPrediction, ClassificationPrediction from typing import Literal, Any from litellm import ChatCompletionRequest

class TaskAnalysisRubric(BaseRubrics): def init(self): super().init( toolname="analyzetask", tooldescription="Analyze task characteristics and complexity", features=[ Feature( name="requiresclarification", description="Task requires additional clarification from user", predictiontype=BinaryPrediction ), Feature( name="complexitylevel", description="Overall complexity assessment", predictiontype=ClassificationPrediction[Literal["simple", "moderate", "complex"]] ) ], systemmessage="You are an expert task analyzer.", user_message="Analyze the following task:" )

def create_annotation_request(self, inputs: dict[str, Any], model: str = "openai/o3-2025-04-16") -> ChatCompletionRequest:
    return {
        "model": model,
        "messages": [
            {"role": "system", "content": self.system_message},
            {"role": "user", "content": f"{self.user_message}\n\n{inputs['task_description']}"}
        ],
        "tools": self.tools,
        "tool_choice": self.tool_choice
    }

```

Tool Schema Generation

```python rubric = TaskAnalysisRubric()

OpenAI-compatible tool schema

print(rubric.tools)

[{"type": "function", "function": {"name": "analyze_task", "parameters": {...}}}]

print(rubric.tool_choice)

{"type": "function", "function": {"name": "analyze_task"}}

```

LLM Integration

```python from critic_rubrics import Annotator from litellm import completion

Single request

rubric = TaskAnalysisRubric() request = rubric.createannotationrequest({"task_description": "Build a web scraper"}) response = Annotator.annotate(request, model="openai/gpt-4")

Extract structured data

toolcall = response.choices[0].message.toolcalls[0] featuredatalist = rubric.toolcalltofeaturedata(tool_call)

for featuredata in featuredatalist: print(f"{featuredata.feature.name}: {featuredata.prediction.todict()}") ```

Batch Processing

```python

Send batch requests

requests = [rubric.createannotationrequest({"taskdescription": task}) for task in tasks] batchids = Annotator.batchannotate( requests, outputdir="./batchresults", customllm_provider="openai", model="openai/gpt-4" )

Retrieve results

for batchid in batchids: status, results = Annotator.getbatchresults(batchid, customllm_provider="openai") if status["status"] == "completed": for result in results: # Process batch result pass ```

Data Flow

1. Define Rubric → 2. Generate Tool Schema → 3. Send to LLM → 4. Parse Response → 5. Typed FeatureData ↓ ↓ ↓ ↓ ↓ BaseRubrics.tools ChatCompletionRequest ModelResponse tool_call_to_feature_data() List[FeatureData]

Key Methods

Prediction Methods

• to_tool_properties(field_name, field_description, rationale_description) → dict[str, Any]
• from_tool_args(feature_name, tool_args) → BasePrediction
• to_dict() → dict[str, Any]

Rubric Methods

• tools → list[ChatCompletionToolParam]
• tool_choice → ChatCompletionToolChoiceObjectParam
• create_annotation_request(inputs, model) → ChatCompletionRequest
• tool_call_to_feature_data(tool_call) → list[FeatureData]

Annotator Methods

• annotate(request, **kwargs) → ModelResponse
• batch_annotate(requests, output_dir, custom_llm_provider, **kwargs) → list[str]
• get_batch_results(batch_id, custom_llm_provider, **kwargs) → tuple[dict, list[dict]]

Installation

```bash

Runtime dependencies

pip install -e .

Development setup

uv sync --group dev ```

Requirements

• Python 3.12+
• pydantic >= 2.11.7
• litellm >= 1.76.0

Example: Complete Workflow

```python from critic_rubrics import BaseRubrics, Feature, BinaryPrediction, Annotator from typing import Any from litellm import ChatCompletionRequest

1. Define rubric

class CodeReviewRubric(BaseRubrics): def init(self): super().init( toolname="reviewcode", tooldescription="Review code for potential issues", features=[ Feature("hasbugs", "Code contains potential bugs", BinaryPrediction), Feature("needsrefactor", "Code needs refactoring", BinaryPrediction) ], systemmessage="You are a senior code reviewer." )

def create_annotation_request(self, inputs: dict[str, Any], model: str = "openai/gpt-4") -> ChatCompletionRequest:
    return {
        "model": model,
        "messages": [
            {"role": "system", "content": self.system_message},
            {"role": "user", "content": f"Review this code:\n\n{inputs['code']}"}
        ],
        "tools": self.tools,
        "tool_choice": self.tool_choice
    }

2. Use rubric

rubric = CodeReviewRubric() request = rubric.createannotationrequest({"code": "def add(a, b): return a + b"}) response = Annotator.annotate(request)

3. Extract results

toolcall = response.choices[0].message.toolcalls[0] features = rubric.toolcalltofeaturedata(tool_call)

for featuredata in features: pred = featuredata.prediction print(f"{feature_data.feature.name}: {pred.detected} - {pred.rationale}") ```