Bayesian Optimization via Prompting (BOPRO)

Official implementation of the ICLR 2025 paper: "Searching for Optimal Solutions with LLMs via Bayesian Optimization" (or BOPRO; Bayesian Optimization via Prompting).

Paper: https://openreview.net/pdf?id=aVfDrl7xDV

Setup

• Install miniconda (https://docs.anaconda.com/miniconda/)
• Create and activate conda environment shell conda create -n bopro \ python=3.11 pandas numpy matplotlib scikit-learn jupyterlab --y \ && conda activate bopro
• Install PyTorch shell pip3 install torch
• Install packages in requirements.txt shell pip3 install -r requirements.txt
• Install SimCSE from source ```shell git clone https://github.com/princeton-nlp/SimCSE.git && cd SimCSE

vi setup.py # remove version numbers from packages (L21 and L24) vi simcse/tool.py # add argument silent=False in L52 vi simcse/tool.py # add disable=silent to the tqdm call in L65

python setup.py install && cd .. && rm -rf SimCSE - Install Laplace botorch shell pip3 install laplace-torch \ && pip3 install git+https://git@github.com/wiseodd/asdl@asdfghjkl \ && pip3 install laplace-bayesopt

# pip3 install git+https://github.com/aleximmer/laplace.git@0.2 \ # && pip3 install laplace-bayesopt \ # && python -m pip3 install setuptools==69.5.1 # needed for laplace-bayesopt ```

Experiments

Semantle

```shell

LogEI

python src/semantlebo.py --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --reprprompt="%s" --lowdimstrategy="off" --acquisitionfn="logEI" --outdir="outputs-semantle-20241020-extended-noscores-matern" --runid="logei" --taskfpath="data/semantle/train" --nevaluations=1000 --nseeds=5 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=10 --vec2textnparallel=2 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores python src/semantlebo.py --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --reprprompt="%s" --lowdimstrategy="off" --acquisitionfn="logEI" --outdir="outputs-semantle-20241020-extended-noscores-matern" --runid="logei" --taskfpath="data/semantle/test" --nevaluations=1000 --nseeds=5 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=10 --vec2textnparallel=2 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores

UCB

python src/semantlebo.py --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --reprprompt="%s" --lowdimstrategy="off" --acquisitionfn="UCB" --outdir="outputs-semantle-20241020-extended-noscores-matern" --runid="ucb" --taskfpath="data/semantle/train" --nevaluations=1000 --nseeds=5 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=10 --vec2textnparallel=2 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores python src/semantlebo.py --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --reprprompt="%s" --lowdimstrategy="off" --acquisitionfn="UCB" --outdir="outputs-semantle-20241020-extended-noscores-matern" --runid="ucb" --taskfpath="data/semantle/test" --nevaluations=1000 --nseeds=5 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=10 --vec2textnparallel=2 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores

Thompson sampling

python src/semantlebo.py --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --reprprompt="%s" --lowdimstrategy="off" --acquisitionfn="thompsonsampling" --outdir="outputs-semantle-20241020-extended-noscores-matern" --runid="thompsonsampling" --taskfpath="data/semantle/train" --nevaluations=1000 --nseeds=5 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=10 --vec2textnparallel=2 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores python src/semantlebo.py --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --reprprompt="%s" --lowdimstrategy="off" --acquisitionfn="thompsonsampling" --outdir="outputs-semantle-20241020-extended-noscores-matern" --runid="thompsonsampling" --taskfpath="data/semantle/test" --nevaluations=1000 --nseeds=5 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=10 --vec2textnparallel=2 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores

Repeated sampling

python src/semantlebo.py --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --reprprompt="%s" --lowdimstrategy="off" --acquisitionfn="none" --outdir="outputs-semantle-20241020-extended-noscores-matern" --runid="none" --taskfpath="data/semantle/train" --nevaluations=1000 --nseeds=5 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=10 --vec2textnparallel=2 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores python src/semantlebo.py --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --reprprompt="%s" --lowdimstrategy="off" --acquisitionfn="none" --outdir="outputs-semantle-20241020-extended-noscores-matern" --runid="none" --taskfpath="data/semantle/test" --nevaluations=1000 --nseeds=5 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=10 --vec2textnparallel=2 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores

Random

python src/semantlebo.py --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --reprprompt="%s" --lowdimstrategy="off" --acquisitionfn="random" --outdir="outputs-semantle-20241020-extended-noscores-matern" --runid="random" --taskfpath="data/semantle/train" --nevaluations=1000 --nseeds=5 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=10 --vec2textnparallel=2 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores python src/semantlebo.py --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --reprprompt="%s" --lowdimstrategy="off" --acquisitionfn="random" --outdir="outputs-semantle-20241020-extended-noscores-matern" --runid="random" --taskfpath="data/semantle/test" --nevaluations=1000 --nseeds=5 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=10 --vec2textnparallel=2 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores ```

Molecule Optimization

```shell

LogEI

python src/moloptbo.py --genmodel="mistral-large-2407" --reprmodel="molformer" --reprprompt="targetbased" --lowdimstrategy="off" --acquisitionfn="logEI" --outdir="outputs-molopt-20241117-mistral" --runid="logei" --taskfpath="data/molopt/data.json" --nevaluations=500 --nseeds=3 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=5 --vec2textnparallel=1 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores --no-visualize_posterior

UCB

python src/moloptbo.py --genmodel="mistral-large-2407" --reprmodel="molformer" --reprprompt="targetbased" --lowdimstrategy="off" --acquisitionfn="UCB" --outdir="outputs-molopt-20241117-mistral" --runid="ucb" --taskfpath="data/molopt/data.json" --nevaluations=500 --nseeds=3 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=5 --vec2textnparallel=1 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores --no-visualize_posterior

Thompson sampling

python src/moloptbo.py --genmodel="mistral-large-2407" --reprmodel="molformer" --reprprompt="targetbased" --lowdimstrategy="off" --acquisitionfn="thompsonsampling" --outdir="outputs-molopt-20241117-mistral" --runid="thompsonsampling" --taskfpath="data/molopt/data.json" --nevaluations=500 --nseeds=3 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=5 --vec2textnparallel=1 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores --no-visualize_posterior

Repeated sampling

python src/moloptbo.py --genmodel="mistral-large-2407" --reprmodel="molformer" --reprprompt="targetbased" --lowdimstrategy="off" --acquisitionfn="none" --outdir="outputs-molopt-20241117-mistral" --runid="none" --taskfpath="data/molopt/data.json" --nevaluations=500 --nseeds=3 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=5 --vec2textnparallel=1 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores --no-visualize_posterior

Random

python src/moloptbo.py --genmodel="mistral-large-2407" --reprmodel="molformer" --reprprompt="targetbased" --lowdimstrategy="off" --acquisitionfn="random" --outdir="outputs-molopt-20241117-mistral" --runid="random" --taskfpath="data/molopt/data.json" --nevaluations=500 --nseeds=3 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=5 --vec2textnparallel=1 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores --no-visualize_posterior

OPRO

python src/moloptbo.py --genmodel="mistral-large-2407" --reprmodel="molformer" --reprprompt="targetbased" --lowdimstrategy="off" --acquisitionfn="OPRO" --outdir="outputs-molopt-20241117-mistral" --runid="opro" --taskfpath="data/molopt/data.json" --nevaluations=500 --nseeds=3 --llmtemperature=1 --optbatchsize=1 --vec2textbatchsize=5 --vec2textnparallel=1 --kernelmeanpriormean=0.4 --kernelmeanpriorstd=0.01 --kernellengthscalepriorconcentration=4 --kernellengthscalepriorrate=2 --kerneloutputscalepriorconcentration=4 --kerneloutputscalepriorrate=2 --gpkernel="matern" --gpnoisevar=0.001 --no-kernelperdimlengthscale --usemethoddefaults --no-arcusescores --no-visualize_posterior ```

1D-ARC-Hard

```shell

UCB

CUDAVISIBLEDEVICES=3 python src/arcbo.py --task="arc" --taskfpath="data/arc-1d-full" --bboxmodel="codegen" --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --candidatesfname="outputs-arc1d-full-mistral/cands100" --outdir="outputs-arc1d-hard-mistral-20240922" --nwarmstart=100 --no-visualizeposterior --llmtokens=2048 --llmtemperature=1 --nseeds=1 --vec2textuniqueretries=0 --vec2textfixretries=1 --vec2textreviseretries=0 --reprcodegenstrategy="docstring" --no-reprcodegeninstruct --no-arcusecode --arcusedocstring --usemethoddefaults --optbatchsize=1 --kernellengthscalepriorconcentration=35 --kernellengthscalepriorrate=30 --kernelmeanpriormean=0.5 --kernelmeanpriorstd=0.01 --nevaluations=200 --vec2textbatchsize=10 --vec2textnparallel=5 --vec2textdemos=3 --no-surrskipinvalidcandidates --setinvalidtozero --runid="ucb" --acquisitionfn="UCB" --target=""

TS

CUDAVISIBLEDEVICES=3 python src/arcbo.py --task="arc" --taskfpath="data/arc-1d-full" --bboxmodel="codegen" --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --candidatesfname="outputs-arc1d-full-mistral/cands100" --outdir="outputs-arc1d-hard-mistral-20240922" --nwarmstart=100 --no-visualizeposterior --llmtokens=2048 --llmtemperature=1 --nseeds=1 --vec2textuniqueretries=0 --vec2textfixretries=1 --vec2textreviseretries=0 --reprcodegenstrategy="docstring" --no-reprcodegeninstruct --no-arcusecode --arcusedocstring --usemethoddefaults --optbatchsize=1 --kernellengthscalepriorconcentration=35 --kernellengthscalepriorrate=30 --kernelmeanpriormean=0.5 --kernelmeanpriorstd=0.01 --nevaluations=200 --vec2textbatchsize=10 --vec2textnparallel=5 --vec2textdemos=3 --no-surrskipinvalidcandidates --setinvalidtozero --runid="thompsonsampling" --acquisitionfn="thompsonsampling" --target=""

OPRO

CUDAVISIBLEDEVICES=3 python src/arcbo.py --task="arc" --taskfpath="data/arc-1d-full" --bboxmodel="codegen" --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --candidatesfname="outputs-arc1d-full-mistral/cands100" --outdir="outputs-arc1d-hard-mistral-20240922" --nwarmstart=100 --no-visualizeposterior --llmtokens=2048 --llmtemperature=1 --nseeds=1 --vec2textuniqueretries=0 --vec2textfixretries=1 --vec2textreviseretries=0 --reprcodegenstrategy="docstring" --no-reprcodegeninstruct --no-arcusecode --arcusedocstring --usemethoddefaults --optbatchsize=1 --kernellengthscalepriorconcentration=35 --kernellengthscalepriorrate=30 --kernelmeanpriormean=0.5 --kernelmeanpriorstd=0.01 --nevaluations=200 --vec2textbatchsize=10 --vec2textnparallel=5 --vec2textdemos=3 --no-surrskipinvalidcandidates --setinvalidtozero --runid="opro" --acquisitionfn="OPRO" --target=""

None

CUDAVISIBLEDEVICES=3 python src/arcbo.py --task="arc" --taskfpath="data/arc-1d-full" --bboxmodel="codegen" --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --candidatesfname="outputs-arc1d-full-mistral/cands100" --outdir="outputs-arc1d-hard-mistral-20240922" --nwarmstart=100 --no-visualizeposterior --llmtokens=2048 --llmtemperature=1 --nseeds=1 --vec2textuniqueretries=0 --vec2textfixretries=1 --vec2textreviseretries=0 --reprcodegenstrategy="docstring" --no-reprcodegeninstruct --no-arcusecode --arcusedocstring --usemethoddefaults --optbatchsize=1 --kernellengthscalepriorconcentration=35 --kernellengthscalepriorrate=30 --kernelmeanpriormean=0.5 --kernelmeanpriorstd=0.01 --nevaluations=200 --vec2textbatchsize=10 --vec2textnparallel=5 --vec2textdemos=3 --no-surrskipinvalidcandidates --setinvalidtozero --runid="none" --acquisitionfn="none" --target=""

BLMX:

UCB

CUDAVISIBLEDEVICES=6 python src/arcbo.py --task="arc" --taskfpath="data/arc-1d-full" --bboxmodel="codegen" --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --candidatesfname="outputs-arc1d-full-mistral/cands100" --outdir="outputs-arc1d-hard-mistral-20240922" --nwarmstart=100 --no-visualizeposterior --llmtokens=2048 --llmtemperature=1 --nseeds=1 --vec2textuniqueretries=0 --vec2textfixretries=1 --vec2textreviseretries=0 --reprcodegenstrategy="docstring" --no-reprcodegeninstruct --no-arcusecode --arcusedocstring --usemethoddefaults --optbatchsize=1 --kernellengthscalepriorconcentration=35 --kernellengthscalepriorrate=30 --kernelmeanpriormean=0.5 --kernelmeanpriorstd=0.01 --nevaluations=200 --vec2textbatchsize=10 --vec2textnparallel=5 --vec2textdemos=3 --no-surrskipinvalidcandidates --setinvalidtozero --runid="lmx-ucb" --acquisitionfn="UCB" --vec2text_lmx --target=""

TS

CUDAVISIBLEDEVICES=3 python src/arcbo.py --task="arc" --taskfpath="data/arc-1d-full" --bboxmodel="codegen" --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --candidatesfname="outputs-arc1d-full-mistral/cands100" --outdir="outputs-arc1d-hard-mistral-20240922" --nwarmstart=100 --no-visualizeposterior --llmtokens=2048 --llmtemperature=1 --nseeds=1 --vec2textuniqueretries=0 --vec2textfixretries=1 --vec2textreviseretries=0 --reprcodegenstrategy="docstring" --no-reprcodegeninstruct --no-arcusecode --arcusedocstring --usemethoddefaults --optbatchsize=1 --kernellengthscalepriorconcentration=35 --kernellengthscalepriorrate=30 --kernelmeanpriormean=0.5 --kernelmeanpriorstd=0.01 --nevaluations=200 --vec2textbatchsize=10 --vec2textnparallel=5 --vec2textdemos=3 --no-surrskipinvalidcandidates --setinvalidtozero --runid="lmx-thompsonsampling" --acquisitionfn="thompsonsampling" --vec2text_lmx --target=""

LMX

CUDAVISIBLEDEVICES=3 python src/arcbo.py --task="arc" --taskfpath="data/arc-1d-full" --bboxmodel="codegen" --genmodel="mistral-large-2407" --reprmodel="gte-qwen-2-1.5b-instruct" --candidatesfname="outputs-arc1d-full-mistral/cands100" --outdir="outputs-arc1d-hard-mistral-20240922" --nwarmstart=100 --no-visualizeposterior --llmtokens=2048 --llmtemperature=1 --nseeds=1 --vec2textuniqueretries=0 --vec2textfixretries=1 --vec2textreviseretries=0 --reprcodegenstrategy="docstring" --no-reprcodegeninstruct --no-arcusecode --arcusedocstring --usemethoddefaults --optbatchsize=1 --kernellengthscalepriorconcentration=35 --kernellengthscalepriorrate=30 --kernelmeanpriormean=0.5 --kernelmeanpriorstd=0.01 --nevaluations=200 --vec2textbatchsize=10 --vec2textnparallel=5 --vec2textdemos=3 --no-surrskipinvalidcandidates --setinvalidtozero --runid="lmx" --acquisitionfn="OPRO" --vec2text_lmx --target="" ```

---

License

This project is licensed under the Apache-2.0 License.

---

✍️ Get in touch!

Please reach out to us on email or open a GitHub issue in case of any issues running the code: dagarwal@cs.umass.edu (Dhruv Agarwal), mghuhan@amazon.com (Manoj Ghuhan Arivazhagan).

---

📄 Citation

If you find our work useful, please cite our paper:

@inproceedings{ agarwal2025searching, title={Searching for Optimal Solutions with {LLM}s via Bayesian Optimization}, author={Dhruv Agarwal and Manoj Ghuhan Arivazhagan and Rajarshi Das and Sandesh Swamy and Sopan Khosla and Rashmi Gangadharaiah}, booktitle={The Thirteenth International Conference on Learning Representations}, year={2025}, url={https://openreview.net/forum?id=aVfDrl7xDV} }