https://github.com/cy-suite/vllm

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
✓
Academic publication links
Links to: arxiv.org
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (8.6%) to scientific vocabulary

Last synced: 9 months ago · JSON representation

Repository

Basic Info

Host: GitHub
Owner: cy-suite
License: apache-2.0
Language: Python
Default Branch: main
Size: 299 MB

Statistics

Stars: 0
Watchers: 0
Forks: 0
Open Issues: 4
Releases: 0

Created over 1 year ago · Last pushed about 1 year ago

Metadata Files

Readme Contributing Funding License Code of conduct Codeowners Security

Easy, fast, and cheap LLM serving for everyone

The first vLLM meetup in 2025 is happening on January 22nd, Wednesday, with Google Cloud in San Francisco! We will talk about vLLM's performant V1 architecture, Q1 roadmap, Google Cloud's innovation around vLLM: networking, Cloud Run, Vertex, and TPU! Register Now

Latest News 🔥 - [2024/12] vLLM joins pytorch ecosystem! Easy, Fast, and Cheap LLM Serving for Everyone! - [2024/11] We hosted the seventh vLLM meetup with Snowflake! Please find the meetup slides from vLLM team here, and Snowflake team here. - [2024/10] We have just created a developer slack (slack.vllm.ai) focusing on coordinating contributions and discussing features. Please feel free to join us there! - [2024/10] Ray Summit 2024 held a special track for vLLM! Please find the opening talk slides from the vLLM team here. Learn more from the talks from other vLLM contributors and users! - [2024/09] We hosted the sixth vLLM meetup with NVIDIA! Please find the meetup slides here. - [2024/07] We hosted the fifth vLLM meetup with AWS! Please find the meetup slides here. - [2024/07] In partnership with Meta, vLLM officially supports Llama 3.1 with FP8 quantization and pipeline parallelism! Please check out our blog post here. - [2024/06] We hosted the fourth vLLM meetup with Cloudflare and BentoML! Please find the meetup slides here. - [2024/04] We hosted the third vLLM meetup with Roblox! Please find the meetup slides here. - [2024/01] We hosted the second vLLM meetup with IBM! Please find the meetup slides here. - [2023/10] We hosted the first vLLM meetup with a16z! Please find the meetup slides here. - [2023/08] We would like to express our sincere gratitude to Andreessen Horowitz (a16z) for providing a generous grant to support the open-source development and research of vLLM. - [2023/06] We officially released vLLM! FastChat-vLLM integration has powered LMSYS Vicuna and Chatbot Arena since mid-April. Check out our blog post.

About

vLLM is a fast and easy-to-use library for LLM inference and serving.

vLLM is fast with:

State-of-the-art serving throughput
Efficient management of attention key and value memory with PagedAttention
Continuous batching of incoming requests
Fast model execution with CUDA/HIP graph
Quantizations: GPTQ, AWQ, INT4, INT8, and FP8.
Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
Speculative decoding
Chunked prefill

Performance benchmark: We include a performance benchmark at the end of our blog post. It compares the performance of vLLM against other LLM serving engines (TensorRT-LLM, SGLang and LMDeploy). The implementation is under nightly-benchmarks folder and you can reproduce this benchmark using our one-click runnable script.

vLLM is flexible and easy to use with:

Seamless integration with popular Hugging Face models
High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
Tensor parallelism and pipeline parallelism support for distributed inference
Streaming outputs
OpenAI-compatible API server
Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
Prefix caching support
Multi-lora support

vLLM seamlessly supports most popular open-source models on HuggingFace, including: - Transformer-like LLMs (e.g., Llama) - Mixture-of-Expert LLMs (e.g., Mixtral, Deepseek-V2 and V3) - Embedding Models (e.g. E5-Mistral) - Multi-modal LLMs (e.g., LLaVA)

Find the full list of supported models here.

Getting Started

Install vLLM with pip or from source:

bash pip install vllm

Visit our documentation to learn more. - Installation - Quickstart - List of Supported Models

Contributing

We welcome and value any contributions and collaborations. Please check out CONTRIBUTING.md for how to get involved.

Citation

If you use vLLM for your research, please cite our paper: bibtex @inproceedings{kwon2023efficient, title={Efficient Memory Management for Large Language Model Serving with PagedAttention}, author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica}, booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles}, year={2023} }

Contact Us

For technical questions and feature requests, please use Github issues or discussions.
For discussing with fellow users, please use Discord.
For coordinating contributions and development, please use Slack.
For security disclosures, please use Github's security advisory feature.
For collaborations and partnerships, please contact us at vllm-questions AT lists.berkeley.edu.

Media Kit

If you wish to use vLLM's logo, please refer to our media kit repo.

Owner

Name: cy-suite
Login: cy-suite
Kind: organization

Repositories: 1
Profile: https://github.com/cy-suite

GitHub Events

Total

Public event: 1

Last Year

Public event: 1

Dependencies

.github/workflows/actionlint.yml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite

.github/workflows/add_label_automerge.yml actions

actions/github-script 60a0d83039c74a4aee543508d2ffcb1c3799cdea composite

.github/workflows/clang-format.yml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
actions/setup-python 0b93645e9fea7318ecaed2b359559ac225c90a2b composite

.github/workflows/cleanup_pr_body.yml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
actions/setup-python 0b93645e9fea7318ecaed2b359559ac225c90a2b composite

.github/workflows/codespell.yml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
actions/setup-python 0b93645e9fea7318ecaed2b359559ac225c90a2b composite

.github/workflows/lint-and-deploy.yaml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
actions/setup-python 0b93645e9fea7318ecaed2b359559ac225c90a2b composite
azure/setup-helm fe7b79cd5ee1e45176fcad797de68ecaf3ca4814 composite
helm/chart-testing-action e6669bcd63d7cb57cb4380c33043eebe5d111992 composite
helm/kind-action 0025e74a8c7512023d06dc019c617aa3cf561fde composite

.github/workflows/mypy.yaml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
actions/setup-python 0b93645e9fea7318ecaed2b359559ac225c90a2b composite

.github/workflows/png-lint.yml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite

.github/workflows/publish.yml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
actions/github-script 60a0d83039c74a4aee543508d2ffcb1c3799cdea composite

.github/workflows/reminder_comment.yml actions

actions/github-script 60a0d83039c74a4aee543508d2ffcb1c3799cdea composite

.github/workflows/ruff.yml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
actions/setup-python 0b93645e9fea7318ecaed2b359559ac225c90a2b composite

.github/workflows/shellcheck.yml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite

.github/workflows/sphinx-lint.yml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
actions/setup-python 0b93645e9fea7318ecaed2b359559ac225c90a2b composite

.github/workflows/stale.yml actions

actions/stale 28ca1036281a5e5922ead5184a1bbf96e5fc984e composite

.github/workflows/yapf.yml actions

actions/checkout 11bd71901bbe5b1630ceea73d27597364c9af683 composite
actions/setup-python 0b93645e9fea7318ecaed2b359559ac225c90a2b composite

Dockerfile docker

base latest build
nvidia/cuda ${CUDA_VERSION}-devel-ubuntu20.04 build
nvidia/cuda ${CUDA_VERSION}-base-ubuntu22.04 build
vllm-base latest build
vllm-openai-base latest build

benchmarks/kernels/requirements.txt pypi

pandas *

docs/requirements-docs.txt pypi

aiohttp *
cloudpickle *
fastapi *
mistral_common >=1.5.0
msgspec *
myst-parser ==3.0.1
openai *
partial-json-parser *
py-cpuinfo *
pydantic >=2.8
requests *
sphinx ==6.2.1
sphinx-argparse ==0.4.0
sphinx-book-theme ==1.0.1
sphinx-copybutton ==0.5.2
sphinx-design ==0.6.1
sphinx-togglebutton ==0.3.2
starlette *
torch *
transformers *
zmq *

pyproject.toml pypi

requirements-build.txt pypi

cmake >=3.26
jinja2 *
ninja *
packaging *
setuptools >=61
setuptools-scm >=8
torch ==2.5.1
wheel *

requirements-common.txt pypi

aiohttp *
blake3 *
cloudpickle *
compressed-tensors ==0.8.1
depyf ==0.18.0
einops *
fastapi >=0.107.0,
fastapi >=0.107.0,<0.113.0
gguf ==0.10.0
importlib_metadata *
lark ==1.2.2
lm-format-enforcer >=0.10.9,<0.11
mistral_common >=1.5.0
msgspec *
numpy <2.0.0
openai >=1.52.0
outlines ==0.1.11
partial-json-parser *
pillow *
prometheus-fastapi-instrumentator >=7.0.0
prometheus_client >=0.18.0
protobuf *
psutil *
py-cpuinfo *
pydantic >=2.9
pyyaml *
pyzmq *
requests >=2.26.0
sentencepiece *
setuptools >=74.1.1
six >=1.16.0
tiktoken >=0.6.0
tokenizers >=0.19.1
tqdm *
transformers >=4.45.2
typing_extensions >=4.10
uvicorn *
xgrammar >=0.1.6

requirements-cpu.txt pypi

datasets *
torch ==2.5.1
torchvision *

requirements-cuda.txt pypi

nvidia-ml-py >=12.560.30
ray >=2.9
torch ==2.5.1
xformers ==0.0.28.post3

requirements-dev.txt pypi

requirements-hpu.txt pypi

pandas *
ray *
setuptools >=61
setuptools-scm >=8
tabulate *
triton *

requirements-lint.txt pypi

clang-format ==18.1.5
codespell ==2.3.0
isort ==5.13.2
mypy ==1.11.1
ruff ==0.6.5
sphinx-lint ==1.0.0
toml ==0.10.2
tomli ==2.0.2
types-PyYAML *
types-requests *
types-setuptools *
yapf ==0.32.0

requirements-neuron.txt pypi

neuronx-cc *
torch-neuronx >=2.5.0
transformers-neuronx >=0.13.0

requirements-openvino.txt pypi

openvino >=2024.4.0
torch ==2.5.1

requirements-rocm.txt pypi

awscli *
boto3 *
botocore *
datasets *
peft *
pytest-asyncio *
ray >=2.10.0
tensorizer >=2.9.0

requirements-test.in pypi

awscli * test
bitsandbytes >=0.45.0 test
buildkite-test-collector ==0.1.9 test
datamodel_code_generator * test
decord * test
einops * test
httpx * test
librosa * test
lm-eval ==0.4.4 test
matplotlib * test
mistral_common >=1.5.0 test
numpy <2.0.0 test
peft * test
pqdm * test
pytest * test
pytest-asyncio * test
pytest-forked * test
pytest-rerunfailures * test
pytest-shard * test
ray ==2.40.0 test
sentence-transformers * test
soundfile * test
tensorizer >=2.9.0 test
timm * test
torch ==2.5.1 test
transformers_stream_generator * test

requirements-test.txt pypi

absl-py ==2.1.0 test
accelerate ==1.0.1 test
aiohappyeyeballs ==2.4.3 test
aiohttp ==3.10.10 test
aiosignal ==1.3.1 test
annotated-types ==0.7.0 test
anyio ==4.6.2.post1 test
argcomplete ==3.5.1 test
attrs ==24.2.0 test
audioread ==3.0.1 test
awscli ==1.35.23 test
bitsandbytes >=0.45.0 test
black ==24.10.0 test
boto3 ==1.35.57 test
botocore ==1.35.57 test
bounded-pool-executor ==0.0.3 test
buildkite-test-collector ==0.1.9 test
certifi ==2024.8.30 test
cffi ==1.17.1 test
chardet ==5.2.0 test
charset-normalizer ==3.4.0 test
click ==8.1.7 test
colorama ==0.4.6 test
contourpy ==1.3.0 test
cupy-cuda12x ==13.3.0 test
cycler ==0.12.1 test
datamodel-code-generator ==0.26.3 test
dataproperty ==1.0.1 test
datasets ==3.0.2 test
decorator ==5.1.1 test
decord ==0.6.0 test
dill ==0.3.8 test
dnspython ==2.7.0 test
docutils ==0.16 test
einops ==0.8.0 test
email-validator ==2.2.0 test
evaluate ==0.4.3 test
fastrlock ==0.8.2 test
filelock ==3.16.1 test
fonttools ==4.54.1 test
frozenlist ==1.5.0 test
fsspec ==2024.9.0 test
genson ==1.3.0 test
h11 ==0.14.0 test
hiredis ==3.0.0 test
httpcore ==1.0.6 test
httpx ==0.27.2 test
huggingface-hub ==0.26.2 test
idna ==3.10 test
inflect ==5.6.2 test
iniconfig ==2.0.0 test
isort ==5.13.2 test
jinja2 ==3.1.4 test
jmespath ==1.0.1 test
joblib ==1.4.2 test
jsonlines ==4.0.0 test
jsonschema ==4.23.0 test
jsonschema-specifications ==2024.10.1 test
kiwisolver ==1.4.7 test
lazy-loader ==0.4 test
libnacl ==2.1.0 test
librosa ==0.10.2.post1 test
llvmlite ==0.43.0 test
lm-eval ==0.4.4 test
lxml ==5.3.0 test
markupsafe ==3.0.2 test
matplotlib ==3.9.2 test
mbstrdecoder ==1.1.3 test
mistral-common ==1.5.1 test
more-itertools ==10.5.0 test
mpmath ==1.3.0 test
msgpack ==1.1.0 test
multidict ==6.1.0 test
multiprocess ==0.70.16 test
mypy-extensions ==1.0.0 test
networkx ==3.2.1 test
nltk ==3.9.1 test
numba ==0.60.0 test
numexpr ==2.10.1 test
numpy ==1.26.4 test
nvidia-cublas-cu12 ==12.4.5.8 test
nvidia-cuda-cupti-cu12 ==12.4.127 test
nvidia-cuda-nvrtc-cu12 ==12.4.127 test
nvidia-cuda-runtime-cu12 ==12.4.127 test
nvidia-cudnn-cu12 ==9.1.0.70 test
nvidia-cufft-cu12 ==11.2.1.3 test
nvidia-curand-cu12 ==10.3.5.147 test
nvidia-cusolver-cu12 ==11.6.1.9 test
nvidia-cusparse-cu12 ==12.3.1.170 test
nvidia-nccl-cu12 ==2.21.5 test
nvidia-nvjitlink-cu12 ==12.4.127 test
nvidia-nvtx-cu12 ==12.4.127 test
opencv-python-headless ==4.10.0.84 test
packaging ==24.1 test
pandas ==2.2.3 test
pathspec ==0.12.1 test
pathvalidate ==3.2.1 test
peft ==0.13.2 test
pillow ==10.4.0 test
platformdirs ==4.3.6 test
pluggy ==1.5.0 test
pooch ==1.8.2 test
portalocker ==2.10.1 test
pqdm ==0.2.0 test
propcache ==0.2.0 test
protobuf ==5.28.3 test
psutil ==6.1.0 test
py ==1.11.0 test
pyarrow ==18.0.0 test
pyasn1 ==0.6.1 test
pybind11 ==2.13.6 test
pycparser ==2.22 test
pydantic ==2.9.2 test
pydantic-core ==2.23.4 test
pyparsing ==3.2.0 test
pytablewriter ==1.2.0 test
pytest ==8.3.3 test
pytest-asyncio ==0.24.0 test
pytest-forked ==1.6.0 test
pytest-rerunfailures ==14.0 test
pytest-shard ==0.1.2 test
python-dateutil ==2.9.0.post0 test
pytz ==2024.2 test
pyyaml ==6.0.2 test
ray ==2.40.0 test
redis ==5.2.0 test
referencing ==0.35.1 test
regex ==2024.9.11 test
requests ==2.32.3 test
rouge-score ==0.1.2 test
rpds-py ==0.20.1 test
rsa ==4.7.2 test
s3transfer ==0.10.3 test
sacrebleu ==2.4.3 test
safetensors ==0.4.5 test
scikit-learn ==1.5.2 test
scipy ==1.13.1 test
sentence-transformers ==3.2.1 test
sentencepiece ==0.2.0 test
six ==1.16.0 test
sniffio ==1.3.1 test
soundfile ==0.12.1 test
soxr ==0.5.0.post1 test
sqlitedict ==2.1.0 test
sympy ==1.13.1 test
tabledata ==1.3.3 test
tabulate ==0.9.0 test
tcolorpy ==0.1.6 test
tenacity ==9.0.0 test
tensorizer ==2.9.0 test
threadpoolctl ==3.5.0 test
tiktoken ==0.7.0 test
timm ==1.0.11 test
tokenizers ==0.21.0 test
torch ==2.5.1 test
torchvision ==0.20.1 test
tqdm ==4.66.6 test
tqdm-multiprocess ==0.0.11 test
transformers ==4.47.0 test
transformers-stream-generator ==0.0.5 test
triton ==3.1.0 test
typepy ==1.3.2 test
typing-extensions ==4.12.2 test
tzdata ==2024.2 test
urllib3 ==1.26.20 test
word2number ==1.1 test
xxhash ==3.5.0 test
yarl ==1.17.1 test
zstandard ==0.23.0 test

requirements-tpu.txt pypi

cmake >=3.26
jax ==0.4.36.dev20241122
jaxlib ==0.4.36.dev20241122
jinja2 *
ninja *
packaging *
ray *
setuptools-scm >=8
torch ==2.6.0.dev20241126
torchvision ==0.20.0.dev20241126
wheel *

requirements-xpu.txt pypi

cmake >=3.26
jinja2 *
ninja *
packaging *
ray >=2.9
setuptools-scm >=8
triton-xpu ==3.0.0b1
wheel *

setup.py pypi

tests/plugins/vllm_add_dummy_model/setup.py pypi

tests/plugins/vllm_add_dummy_platform/setup.py pypi

tests/vllm_test_utils/setup.py pypi

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science