😺 Agentless

😽News | 🤖LLM Integration | 🐈Setup | 🧶Comparison | 🐈‍⬛Artifacts | 📝Citation | 😻Acknowledgement

😽 News

• Dec 2nd, 2024: We integrated Agentless with Claude 3.5 Sonnet to achieve 40.7% and 50.8% solve rate on SWE-bench lite and verified
• Oct 28th, 2024: We just released OpenAutoCoder-Agentless 1.5!
• July 1st, 2024: We just released OpenAutoCoder-Agentless 1.0! Agentless currently is the best open-source approach on SWE-bench lite with 82 fixes (27.3%) and costing on average $0.34 per issue.

😺 About

Agentless is an agentless approach to automatically solve software development problems. To solve each issue, Agentless follows a simple three phase process: localization, repair, and patch validation. - 🙀 Localization: Agentless employs a hierarchical process to first localize the fault to specific files, then to relevant classes or functions, and finally to fine-grained edit locations - 😼 Repair: Agentless takes the edit locations and samples multiple candidate patches per bug in a simple diff format - 😸 Patch Validation: Agentless selects the regression tests to run and generates additional reproduction test to reproduce the original error. Using the test results, Agentless re-ranks all remaining patches to selects one to submit

🤖 LLM Integration

Agentless uses LiteLLM to provide a unified interface to multiple LLM providers. This integration supports:

• Multiple providers (OpenAI, Anthropic, DeepSeek, etc.)
• Async and streaming operations
• Model-specific features (function calling, reasoning effort, etc.)
• Comprehensive error handling and retries
• TOML-based configuration

For detailed information about the LiteLLM integration, see LiteLLM Integration Documentation.

🐈 Setup

First create the environment

```shell git clone https://github.com/OpenAutoCoder/Agentless.git cd Agentless

conda create -n agentless python=3.11 conda activate agentless pip install -r requirements.txt export PYTHONPATH=$PYTHONPATH:$(pwd) ```

Then export your OpenAI API key shell export OPENAI_API_KEY={key_here}

Now you are ready to run Agentless on the problems in SWE-bench!

[!NOTE]

To reproduce the full SWE-bench lite experiments and follow our exact setup as described in the paper. Please see this README

🧶 Comparison

Below shows the comparison graph between Agentless and the best open-source agent-based approaches on SWE-bench lite

🐈‍⬛ Artifacts

You can download the complete artifacts of Agentless in our v1.5.0 release: - 🐈‍⬛ agentlessswebenchlite: complete Agentless run on SWE-bench Lite - 🐈‍⬛ agentlessswebenchverified: complete Agentless run on SWE-bench Verified - 🐈‍⬛ swebenchrepostructure: preprocessed structure information for each SWE-Bench problem

You can also checkout classification/ folder to obtain our manual classifications of SWE-bench-lite as well as our filtered SWE-bench-lite-S problems.

📝 Citation

bibtex @article{agentless, author = {Xia, Chunqiu Steven and Deng, Yinlin and Dunn, Soren and Zhang, Lingming}, title = {Agentless: Demystifying LLM-based Software Engineering Agents}, year = {2024}, journal = {arXiv preprint}, }

[!NOTE]

The first two authors contributed equally to this work, with author order determined via Nigiri

😻 Acknowledgement

• SWE-bench
• Aider
• SWE-bench-docker