https://github.com/bigbuildbench/camel-ai_crab
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (14.8%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: BigBuildBench
- Language: Python
- Default Branch: master
- Size: 3.97 MB
Statistics
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
🦀 CRAB: Cross-platform Agent Benchmark for Multimodal Embodied Language Model Agents
Documentation | Website & Demos | Blog | Chinese Blog | CAMEL-AI
Overview
CRAB is a framework for building LLM agent benchmark environments in a Python-centric way.
Key Features
🌐 Cross-platform and Multi-environment * Create build agent environments that support various deployment options including in-memory, Docker-hosted, virtual machines, or distributed physical machines, provided they are accessible via Python functions. * Let the agent access all the environments in the same time through a unified interface.
⚙ ️Easy-to-use Configuration
* Add a new action by simply adding a @action decorator on a Python function.
* Define the environment by integrating several actions together.
📐 Novel Benchmarking Suite * Define tasks and the corresponding evaluators in an intuitive Python-native way. * Introduce a novel graph evaluator method providing fine-grained metrics.
Installation
Prerequisites
- Python 3.10 or newer
bash
pip install crab-framework[client]
Experiment on CRAB-Benchmark-v0
All datasets and experiment code are in crab-benchmark-v0 directory. Please carefully read the benchmark tutorial before using our benchmark.
Examples
Run template environment with openai agent
bash
export OPENAI_API_KEY=<your api key>
python examples/single_env.py
python examples/multi_env.py
Demo Video
Cite
Please cite our paper if you use anything related in your work:
@misc{xu2024crab,
title={CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents},
author={Tianqi Xu and Linyao Chen and Dai-Jie Wu and Yanjun Chen and Zecheng Zhang and Xiang Yao and Zhiqiang Xie and Yongchao Chen and Shilong Liu and Bochen Qian and Philip Torr and Bernard Ghanem and Guohao Li},
year={2024},
eprint={2407.01511},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2407.01511},
}
Owner
- Name: BigBuildBench
- Login: BigBuildBench
- Kind: organization
- Repositories: 1
- Profile: https://github.com/BigBuildBench
abbr. B3, benchmarking the repo-level understanding capability of your LLMs by reconstructing project build-file.
GitHub Events
Total
- Create event: 4
Last Year
- Create event: 4
Dependencies
- abatilo/actions-poetry v2 composite
- actions/cache/restore v3 composite
- actions/cache/save v3 composite
- actions/setup-python v3 composite
- ./.github/actions/crab_install * composite
- actions/checkout v3 composite
- peaceiris/actions-gh-pages v3 composite
- JRubics/poetry-publish v1.17 composite
- actions/checkout v3 composite
- actions/create-release v1 composite
- actions/upload-release-asset v1 composite
- ./.github/actions/crab_install * composite
- actions/checkout v3 composite
- certifi ^2024.2.2 develop
- ipykernel ^6.29.3 develop
- mypy ^1.8.0 develop
- myst-parser ^4 develop
- pandas ^2.2.2 develop
- pre-commit ^3.7.0 develop
- pytest ^8.0.0 develop
- ruff ^0.6.5 develop
- sphinx ^7 develop
- sphinx-book-theme * develop
- python ^3.10, <3.12