https://github.com/google-deepmind/opro
official code for "Large Language Models as Optimizers"
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.5%) to scientific vocabulary
Repository
official code for "Large Language Models as Optimizers"
Basic Info
- Host: GitHub
- Owner: google-deepmind
- License: apache-2.0
- Language: Python
- Default Branch: main
- Homepage: https://arxiv.org/abs/2309.03409
- Size: 4.51 MB
Statistics
- Stars: 572
- Watchers: 7
- Forks: 71
- Open Issues: 6
- Releases: 0
Metadata Files
README.md
Large Language Models as Optimizers
This repository contains the code for the paper
Large Language Models as Optimizers\ Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen [* Equal Contribution]\ arXiv: 2309.03409
Dependency requirements
The code has been verified to work under Python 3.10.13 with the following dependencies:
- absl-py (2.0.0)
- google.generativeai (0.1.0)
- immutabledict (3.0.0)
- openai (0.27.2)
Usage
Prompt optimization
Use opro/optimization/optimize_instructions.py, follow the steps at the top.
A quickstarter:
python optimize_instructions.py --optimizer="gpt-3.5-turbo" --scorer="text-bison"
--instruction_pos="Q_begin" --dataset="gsm8k" --task="train" --palm_api_key="<your_palm_api_key>" --openai_api_key="<your_openai_api_key>"
Prompt evaluation
Use opro/evaluation/evaluate_instructions.py, follow the steps at the top.
A quickstarter:
python evaluate_instructions.py --scorer="text-bison" --dataset="gsm8k" --task="test" --instruction_pos="Q_begin" --evaluate_training_fold=false --evaluate_test_fold=true --palm_api_key="<your_palm_api_key>"
Linear regression
Use opro/optimization/optimize_linear_regression.py, follow the steps at the top.
Traveling salesman problem
Use opro/optimization/optimize_tsp.py, follow the steps at the top.
Supported models
The code in this repository currently supports text-bison and GPT models. Alternatively, you may serve your own model and plug it in here, similar to the existing prompting APIs in opro/prompt_utils.py.
Precaution on API costs
Calling the PaLM or GPT APIs for prompt optimization and evaluation may incur unexpectedly large costs. Please carefully estimate the cost and/or start with lighter use (e.g., evaluate on a smaller portion of the benchmark dataset or run optimization for fewer steps) before the formal experimentations, or prompt self-served models instead.
Citation
If you have used our code in your research, please cite our paper:
@article{yang2023large,
title={Large language models as optimizers},
author={Yang, Chengrun and Wang, Xuezhi and Lu, Yifeng and Liu, Hanxiao and Le, Quoc V and Zhou, Denny and Chen, Xinyun},
journal={arXiv preprint arXiv:2309.03409},
year={2023}
}
Disclaimer: this is not an officially supported Google product.
Owner
- Name: Google DeepMind
- Login: google-deepmind
- Kind: organization
- Website: https://www.deepmind.com/
- Repositories: 245
- Profile: https://github.com/google-deepmind
GitHub Events
Total
- Issues event: 3
- Watch event: 180
- Issue comment event: 2
- Pull request event: 1
- Fork event: 43
Last Year
- Issues event: 3
- Watch event: 179
- Issue comment event: 2
- Pull request event: 1
- Fork event: 43
Committers
Last synced: 10 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| chengrunyang | y****3@g****m | 12 |
Issues and Pull Requests
Last synced: 10 months ago
All Time
- Total issues: 9
- Total pull requests: 2
- Average time to close issues: 8 days
- Average time to close pull requests: N/A
- Total issue authors: 9
- Total pull request authors: 2
- Average comments per issue: 1.44
- Average comments per pull request: 0.5
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 4
- Pull requests: 2
- Average time to close issues: 36 minutes
- Average time to close pull requests: N/A
- Issue authors: 4
- Pull request authors: 2
- Average comments per issue: 0.25
- Average comments per pull request: 0.5
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- georgewanglz2019 (1)
- wac81 (1)
- Victordongy (1)
- hhycoding (1)
- luochenxin (1)
- chansonzhang (1)
- gautamjajoo (1)
- joyce0105-ops (1)
- datalee (1)
Pull Request Authors
- JGalego (2)
- gautamjajoo (2)