Science Score: 54.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.5%) to scientific vocabulary
Repository
Core code for AgentDojo
Basic Info
- Host: GitHub
- Owner: ethz-spylab
- License: mit
- Language: Python
- Default Branch: main
- Size: 314 KB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 3
Metadata Files
README.md
Quickstart
bash
pip install agentdojo
[!IMPORTANT] Note that the API of the package is still under development and might change in the future.
If you want to use the prompt injection detector, you need to install the transformers extra:
bash
pip install "agentdojo[transformers]"
Running the benchmark
The benchmark can be run with the benchmark script. Documentation on how to use the script can be obtained with the --help flag.
For example, to run the workspace suite on the tasks 0 and 1, with gpt-4o-2024-05-13 as the LLM, the tool filter as a defense, and the attack with tool knowlege, run the following command:
bash
python -m agentdojo.scripts.benchmark -s workspace -ut user_task_0 \
-ut user_task_1 --model gpt-4o-2024-05-13 \
--defense tool_filter --attack tool_knowledge
To run the above, but on all suites and tasks, run the following:
bash
python -m agentdojo.scripts.benchmark --model gpt-4o-2024-05-13 \
--defense tool_filter --attack tool_knowledge
Inspect the results
To inspect the results, go to the dedicated results page of the documentation. AgentDojo results are also listed in the Invariant Benchmark Registry.Agent
Documentation of the Dojo
Take a look at our documentation.
Development set-up
Take a look at the development set-up docs.
Citing
If you use AgentDojo in your research, please consider citing our paper:
bibtex
@inproceedings{
debenedetti2024agentdojo,
title={AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for {LLM} Agents},
author={Edoardo Debenedetti and Jie Zhang and Mislav Balunovic and Luca Beurer-Kellner and Marc Fischer and Florian Tram{\`e}r},
booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2024},
url={https://openreview.net/forum?id=m1YYAQjO3w}
}
Owner
- Name: SPY Lab
- Login: ethz-spylab
- Kind: organization
- Location: Switzerland
- Website: https://spylab.ai
- Repositories: 1
- Profile: https://github.com/ethz-spylab
Secure and Private AI research at ETH Zürich
Citation (CITATION.bib)
@inproceedings{
debenedetti2024agentdojo,
title={AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for {LLM} Agents},
author={Edoardo Debenedetti and Jie Zhang and Mislav Balunovic and Luca Beurer-Kellner and Marc Fischer and Florian Tram{\`e}r},
booktitle={The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2024},
url={https://openreview.net/forum?id=m1YYAQjO3w}
}
GitHub Events
Total
- Release event: 2
- Push event: 6
- Create event: 4
Last Year
- Release event: 2
- Push event: 6
- Create event: 4
Packages
- Total packages: 1
-
Total downloads:
- pypi 46 last-month
- Total dependent packages: 0
- Total dependent repositories: 0
- Total versions: 3
- Total maintainers: 1
pypi.org: agentdojo-core
Core code for AgentDojo
- Documentation: https://agentdojo-core.readthedocs.io/
- License: MIT License
-
Latest release: 0.0.3
published 10 months ago
Rankings
Maintainers (1)
Dependencies
- actions/checkout v4 composite
- astral-sh/setup-uv v5 composite
- actions/checkout v4 composite
- astral-sh/setup-uv v5 composite
- pypa/gh-action-pypi-publish release/v1 composite
- anthropic >=0.47.0
- click >=8.1.7
- cohere >=5.3.4
- deepdiff >=7.0.1
- docstring-parser >=0.15
- google-genai >=1.15.0
- langchain >=0.1.17
- openai >=1.59.7
- pydantic [email]>=2.7.1
- python-dotenv >=1.0.1
- pyyaml >=6.0.1
- rich >=13.7.1
- tenacity >=8.2.3
- typing-extensions >=4.11.0
- 124 dependencies