llm-multi-backend-container

Use llama-swap inside a container with vllm, llama.cpp, and exllamav2+tabbyAPI.

Adapting for other machines

This repo is my working config, it's mainly used on a 16 core Ryzen machine with 64 GiB RAM and a single RTX 3090. For customization, you might want to grep for a few keywords: console $ git grep -E '\b868[6-8]\b' # port numbers $ git grep sk-empty # API token/key $ git grep -iE '(logging|loglevel|verbos)' $ git grep -E "\b(8\.6|86)\b" # CUDA compute arch, 8.6 == ampere (RTX 3090)

Usage

console $ head ./bin/host-llm-multi-backend-container.sh $ ./bin/host-llm-multi-backend-container.sh --build --force-recreate See what model/backend combinations are available: console $ curl -s -X GET -H "Authorization: Bearer sk-empty" http://localhost:8686/v1/models | jq -r '.data[].id' | grep -i 'qwen2.5-coder-7b' vllm-Qwen2.5-Coder-7B llamacpp-Qwen2.5-Coder-7B exllamav2-Qwen2.5-Coder-7B

Testing

console $ bash -x scripts/test-chat-completions.sh + curl -s -X POST http://localhost:8688/v1/chat/completions -H 'Content-Type: application/json' -H 'Authorization: Bearer sk-empty' -d '{"model": "llamacpp-glm-4.5-air", "messages": [{"role": "user", "content": "Answer only with the missing word: The capital of Sweden is"}]}' + jq '.choices[0].message.content' "\n<think>We are to answer with only the missing word. The question is: \"The capital of Sweden is\"\n The capital of Sweden is Stockholm. Therefore, the missing word is \"Stockholm\".</think>Stockholm" + retcode=0 + '[' 0 -ne 0 ']' + return 0 + exit

Monitoring

console $ ./scripts/enter-container-llama-swap.sh watch "ps aux | grep -E '(vllm|llama-|tabbyAPI)' | grep -v emacs | grep -v 'grep -E'" $ while true; do clear; date; echo -n "currently loaded model: "; curl -s localhost:8686/running | jq -r '.running[0].model'; echo '...sleeping for 60 seconds'; sleep 60; done $ curl -s localhost:8686/logs/stream/upstream $ curl -s localhost:8686/logs/stream/proxy $ ./scripts/enter-container-llama-swap.sh tail -F /tmp/llama-server-stdout-stderr.log

Working directly with underlying end-point:

console $ curl -H "Authorization: Bearer sk-empty" http://localhost:8686/upstream/llamacpp-Qwen3-30B-A3B/health $ curl -H "Authorization: Bearer sk-empty" http://localhost:8686/upstream/llamacpp-Qwen3-30B-A3B/slots | jq

Downloading models

Downloading Unsloth's Maverick: console $ HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF --exclude "*.gguf" $ HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF --include "UD-Q2_K_XL/*.gguf" Downloading Unsloth's Q2KXL quants (248 GB) of DeepSeek V3 0324: console $ HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download unsloth/DeepSeek-V3-0324-GGUF --exclude "*.gguf" \ && HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download unsloth/DeepSeek-V3-0324-GGUF --include "UD-Q2_K_XL/*.gguf"

Unused configurations

Tidbits