bizCon: Business Conversation Evaluation Framework for LLMs

📋 Table of Contents

🎯 Overview

bizCon is a specialized evaluation framework designed to benchmark Large Language Models (LLMs) on realistic business conversation scenarios. Unlike generic benchmarks, bizCon focuses on practical business use cases involving professional communication, tool integration, and domain-specific knowledge.

Why bizCon?

• Business-Focused: Evaluates models on real-world business scenarios
• Multi-Dimensional: Assesses 5 key aspects of business communication
• Tool Integration: Tests models' ability to use business tools effectively
• Comparative Analysis: Benchmark multiple models side-by-side
• Enterprise-Ready: Professional reporting and analysis capabilities

✨ Key Features

🎭 Diverse Business Scenarios

• Product Inquiries: Enterprise software consultations
• Technical Support: Complex troubleshooting and API integration
• Contract Negotiation: SaaS agreements and enterprise deals
• Appointment Scheduling: Multi-stakeholder coordination
• Compliance Inquiries: Regulatory and data privacy questions
• Implementation Planning: Software deployment strategies
• Service Complaints: Customer service and dispute resolution
• Multi-Department: Cross-functional project coordination

📊 Comprehensive Evaluation Metrics

1. Response Quality (25%) - Factual accuracy and completeness
2. Business Value (25%) - Strategic insight and actionable recommendations
3. Communication Style (20%) - Professionalism and tone appropriateness
4. Tool Usage (20%) - Effective integration with business tools
5. Performance (10%) - Response time and efficiency

🛠️ Business Tool Ecosystem

• Knowledge Base Search
• Product Catalog Lookup
• Pricing Calculator
• Appointment Scheduler
• Customer History Access
• Document Retrieval
• Order Management
• Support Ticket System

🤖 Multi-Model Support

🚀 Quick Start

Installation

```bash

Clone the repository

git clone https://github.com/Olib-AI/bizcon.git cd bizcon

Basic installation

pip install -e .

Install with advanced visualization features (use quotes for zsh)

pip install -e ".[advanced]"

Install all optional features

pip install -e ".[all]" ```

Basic Usage

1. Set up your API keys: bash export OPENAI_API_KEY="your-openai-key" export ANTHROPIC_API_KEY="your-anthropic-key" export MISTRAL_API_KEY="your-mistral-key"

2. Run a quick test: ```bash

   🚀 Test without API keys (uses mock models)

   python test_framework.py

🧪 Run unit and integration tests

python -m pytest tests/

🤖 Test with real models (requires API keys)

python testwithreal_models.py ```

1. Run a benchmark: ```bash # 📊 Compare models on specific scenarios python run.py --scenarios productinquiry001 support_001 --verbose

🏃 Run full benchmark with custom config

python run.py --config config/models.yaml --output results/

💻 Using CLI interface directly

bizcon run --config config/models.yaml --output results/ ```

1. Explore available options: ```bash # 📋 List all available scenarios python run.py --list-scenarios # or: bizcon list-scenarios

🤖 List supported models

python run.py --list-models

or: bizcon list-models

```

Configuration

Customize your evaluation in config/models.yaml:

yaml models: - provider: openai name: gpt-4 temperature: 0.7 max_tokens: 2048 - provider: anthropic name: claude-3-sonnet temperature: 0.7 max_tokens: 2048

Adjust evaluation settings in config/evaluation.yaml:

yaml evaluation: parallel: true num_runs: 3 evaluator_weights: response_quality: 0.25 business_value: 0.25 communication_style: 0.20 tool_usage: 0.20 performance: 0.10

📖 Documentation

Project Structure

bizcon/ ├── config/ # Configuration files │ ├── models.yaml # Model configurations │ └── evaluation.yaml # Evaluation settings ├── core/ # Core evaluation pipeline │ ├── pipeline.py # Main evaluation orchestrator │ └── runner.py # Scenario execution engine ├── models/ # LLM provider integrations │ ├── openai.py # OpenAI client │ ├── anthropic.py # Anthropic client │ └── mistral.py # Mistral AI client ├── scenarios/ # Business conversation scenarios │ ├── product_inquiry.py │ ├── technical_support.py │ └── contract_negotiation.py ├── evaluators/ # Evaluation metrics │ ├── response_quality.py │ ├── business_value.py │ └── communication_style.py ├── tools/ # Business tool implementations │ ├── knowledge_base.py │ ├── scheduler.py │ └── product_catalog.py ├── visualization/ # Advanced visualization and reporting │ ├── charts.py # Static matplotlib charts │ ├── interactive_charts.py # Interactive Plotly charts │ ├── dashboard.py # Basic Flask dashboard │ ├── advanced_dashboard.py # Advanced dashboard with filtering │ ├── analysis_utils.py # Statistical analysis tools │ └── report.py # Report generation └── data/ # Sample business data ├── knowledge_base/ ├── products/ └── pricing/

Creating Custom Scenarios

```python from scenarios.base import BusinessScenario

class CustomBusinessScenario(BusinessScenario): def init(self, scenarioid=None): super().init( scenarioid=scenarioid or "custom001", name="Custom Business Scenario", description="Your custom scenario description", industry="technology", complexity="medium", toolsrequired=["knowledgebase", "scheduler"] )

def _initialize_conversation(self):
    return [{
        "user_message": "Your initial customer message",
        "expected_tool_calls": [
            {"tool_id": "knowledge_base", "parameters": {"query": "example"}}
        ]
    }]

def _initialize_ground_truth(self):
    return {
        "expected_facts": ["Key fact 1", "Key fact 2"],
        "business_objective": "Help customer achieve X",
        "expected_tone": "professional"
    }

```

Adding Custom Evaluators

```python from evaluators.base import BaseEvaluator

class CustomEvaluator(BaseEvaluator): def init(self, weight=1.0): super().init(name="Custom Evaluator", weight=weight)

def evaluate(self, response, scenario, turn_index, conversation_history, tool_calls):
    # Your evaluation logic here
    score = self.calculate_score(response)
    return {
        "score": score,
        "explanation": "Detailed explanation of the score",
        "max_possible": 10.0
    }

```

📊 Sample Results

🏗️ Advanced Usage

Parallel Evaluation

```bash

Run multiple scenarios in parallel

python run.py --scenarios productinquiry001 support001 contract001 --parallel

Or using CLI directly

bizcon run --scenarios productinquiry001 support_001 --parallel ```

Custom Model Parameters

yaml models: - provider: openai name: gpt-4 temperature: 0.3 max_tokens: 1024 parameters: seed: 42 top_p: 0.9

Advanced Visualization Dashboard

```bash

Install advanced features first (use quotes for zsh)

pip install -e ".[advanced]"

Launch interactive dashboard with advanced features

python examples/advanceddashboarddemo.py --results-dir output/

Launch on custom host/port with auto-refresh

python examples/advanceddashboarddemo.py --host 0.0.0.0 --port 8080

Disable auto-refresh for static analysis

python examples/advanceddashboarddemo.py --no-auto-refresh ```

Note: Advanced visualization features require additional dependencies (Plotly, Flask, SciPy). Install with pip install "bizcon[advanced]" (quotes required for zsh) to enable these features.

Scenario Categories

```bash

Run all product inquiry scenarios

python run.py --scenarios productinquiry*

Run scenarios by complexity

python run.py --scenarios complex_* ```

🤝 Contributing

We welcome contributions from the community! Here's how you can help:

Ways to Contribute

• 🐛 Report Bugs: Open an issue with detailed reproduction steps
• ✨ Suggest Features: Propose new scenarios, evaluators, or tools
• 📝 Improve Documentation: Help make our docs clearer
• 🔧 Submit Code: Fix bugs or add new features
• 🧪 Add Test Cases: Improve our test coverage

Development Setup

```bash git clone https://github.com/Olib-AI/bizcon.git cd bizcon pip install -e .

Run framework validation (no API keys needed)

python test_framework.py

Run full test suite

python -m pytest tests/ ```

Contribution Guidelines

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests for new functionality
5. Run the test suite (pytest)
6. Commit your changes (git commit -m 'Add amazing feature')
7. Push to the branch (git push origin feature/amazing-feature)
8. Open a Pull Request

🧪 Testing & Validation

🎯 Framework Validation Status

Running Tests

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

💬 Community

• Website: www.olib.ai
• GitHub: github.com/Olib-AI
• Issues: Report bugs or request features
• Discussions: Join the conversation

👥 Authors

Akram Hasan Sharkar - Author & Lead Developer
Maya Msahal - Co-Author & Research Contributor

Developed at Olib AI

📖 Research Paper

A detailed research paper describing the methodology, evaluation framework, and empirical results of bizCon will be published on arXiv.org. The paper link will be available here upon publication.

Citation format will be provided once the paper is published.

🙏 Acknowledgments

• Built with ❤️ by Akram Hasan Sharkar and Maya Msahal at Olib AI
• Inspired by the need for better business-focused LLM evaluation
• Thanks to all contributors who help make this project better

📈 Roadmap