mosec

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

https://github.com/mosecorg/mosec

Science Score: 54.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
✓
Committers with academic emails
1 of 14 committers (7.1%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (18.4%) to scientific vocabulary

Keywords

cv deep-learning gpu hacktoberfest jax llm llm-serving machine-learning machine-learning-platform mlops model-serving mxnet nerual-network python pytorch rust tensorflow tts

Keywords from Contributors

transformer cryptocurrency diffusion vlm distributed embedded optimism interactive audio deepseek

Last synced: 6 months ago · JSON representation ·

Repository

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

Basic Info

Host: GitHub
Owner: mosecorg
License: apache-2.0
Language: Python
Default Branch: main
Homepage: https://mosecorg.github.io/mosec/
Size: 1.28 MB

Statistics

Stars: 853
Watchers: 12
Forks: 61
Open Issues: 16
Releases: 51

Topics

cv deep-learning gpu hacktoberfest jax llm llm-serving machine-learning machine-learning-platform mlops model-serving mxnet nerual-network python pytorch rust tensorflow tts

Created almost 5 years ago · Last pushed 6 months ago

Metadata Files

Readme Contributing License Citation Codeowners

README.md

MOSEC

Model Serving made Efficient in the Cloud.

Introduction

MOSEC

Mosec is a high-performance and flexible model serving framework for building ML model-enabled backend and microservices. It bridges the gap between any machine learning models you just trained and the efficient online service API.

Highly performant: web layer and task coordination built with Rust 🦀, which offers blazing speed in addition to efficient CPU utilization powered by async I/O
Ease of use: user interface purely in Python 🐍, by which users can serve their models in an ML framework-agnostic manner using the same code as they do for offline testing
Dynamic batching: aggregate requests from different users for batched inference and distribute results back
Pipelined stages: spawn multiple processes for pipelined stages to handle CPU/GPU/IO mixed workloads
Cloud friendly: designed to run in the cloud, with the model warmup, graceful shutdown, and Prometheus monitoring metrics, easily managed by Kubernetes or any container orchestration systems
Do one thing well: focus on the online serving part, users can pay attention to the model optimization and business logic

Installation

Mosec requires Python 3.7 or above. Install the latest PyPI package for Linux or macOS with:

```shell pip install -U mosec

or install with conda

conda install conda-forge::mosec

or install with pixi

pixi add mosec ```

To build from the source code, install Rust and run the following command:

shell make package

You will get a mosec wheel file in the dist folder.

Usage

We demonstrate how Mosec can help you easily host a pre-trained stable diffusion model as a service. You need to install diffusers and transformers as prerequisites:

shell pip install --upgrade diffusers[torch] transformers

Write the server

Click me for server codes with explanations.

Firstly, we import the libraries and set up a basic logger to better observe what happens. ```python from io import BytesIO from typing import List import torch # type: ignore from diffusers import StableDiffusionPipeline # type: ignore from mosec import Server, Worker, get_logger from mosec.mixin import MsgpackMixin logger = get_logger() ``` Then, we **build an API** for clients to query a text prompt and obtain an image based on the [stable-diffusion-v1-5 model](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) in just 3 steps. 1) Define your service as a class which inherits `mosec.Worker`. Here we also inherit `MsgpackMixin` to employ the [msgpack](https://msgpack.org/index.html) serialization format^(a). 2) Inside the `__init__` method, initialize your model and put it onto the corresponding device. Optionally you can assign `self.example` with some data to warm up^(b) the model. Note that the data should be compatible with your handler's input format, which we detail next. 3) Override the `forward` method to write your service handler^(c), with the signature `forward(self, data: Any | List[Any]) -> Any | List[Any]`. Receiving/returning a single item or a tuple depends on whether [dynamic batching](#configuration)^(d) is configured. ```python class StableDiffusion(MsgpackMixin, Worker): def __init__(self): self.pipe = StableDiffusionPipeline.from_pretrained( "sd-legacy/stable-diffusion-v1-5", torch_dtype=torch.float16 ) self.pipe.enable_model_cpu_offload() self.example = ["useless example prompt"] * 4 # warmup (batch_size=4) def forward(self, data: List[str]) -> List[memoryview]: logger.debug("generate images for %s", data) res = self.pipe(data) logger.debug("NSFW: %s", res[1]) images = [] for img in res[0]: dummy_file = BytesIO() img.save(dummy_file, format="JPEG") images.append(dummy_file.getbuffer()) return images ``` > [!NOTE] > > (a) In this example we return an image in the binary format, which JSON does not support (unless encoded with base64 that makes the payload larger). Hence, msgpack suits our need better. If we do not inherit `MsgpackMixin`, JSON will be used by default. In other words, the protocol of the service request/response can be either msgpack, JSON, or any other format (check our [mixins](https://mosecorg.github.io/mosec/reference/interface.html#module-mosec.mixin)). > > (b) Warm-up usually helps to allocate GPU memory in advance. If the warm-up example is specified, the service will only be ready after the example is forwarded through the handler. However, if no example is given, the first request's latency is expected to be longer. The `example` should be set as a single item or a tuple depending on what `forward` expects to receive. Moreover, in the case where you want to warm up with multiple different examples, you may set `multi_examples` (demo [here](https://mosecorg.github.io/mosec/examples/jax.html)). > > (c) This example shows a single-stage service, where the `StableDiffusion` worker directly takes in client's prompt request and responds the image. Thus the `forward` can be considered as a complete service handler. However, we can also design a multi-stage service with workers doing different jobs (e.g., downloading images, model inference, post-processing) in a pipeline. In this case, the whole pipeline is considered as the service handler, with the first worker taking in the request and the last worker sending out the response. The data flow between workers is done by inter-process communication. > > (d) Since dynamic batching is enabled in this example, the `forward` method will wishfully receive a _list_ of string, e.g., `['a cute cat playing with a red ball', 'a man sitting in front of a computer', ...]`, aggregated from different clients for _batch inference_, improving the system throughput. Finally, we append the worker to the server to construct a *single-stage* workflow (multiple stages can be [pipelined](https://en.wikipedia.org/wiki/Pipeline_(computing)) to further boost the throughput, see [this example](https://mosecorg.github.io/mosec/examples/pytorch.html#computer-vision)), and specify the number of processes we want it to run in parallel (`num=1`), and the maximum batch size (`max_batch_size=4`, the maximum number of requests dynamic batching will accumulate before timeout; timeout is defined with the `max_wait_time=10` in milliseconds, meaning the longest time Mosec waits until sending the batch to the Worker). ```python if __name__ == "__main__": server = Server() # 1) `num` specifies the number of processes that will be spawned to run in parallel. # 2) By configuring the `max_batch_size` with the value > 1, the input data in your # `forward` function will be a list (batch); otherwise, it's a single item. server.append_worker(StableDiffusion, num=1, max_batch_size=4, max_wait_time=10) server.run() ```

Run the server

Click me to see how to run and query the server.

The above snippets are merged in our example file. You may directly run at the project root level. We first have a look at the _command line arguments_ (explanations [here](https://mosecorg.github.io/mosec/reference/arguments.html)): ```shell python examples/stable_diffusion/server.py --help ``` Then let's start the server with debug logs: ```shell python examples/stable_diffusion/server.py --log-level debug --timeout 30000 ``` Open `http://127.0.0.1:8000/openapi/swagger/` in your browser to get the OpenAPI doc. And in another terminal, test it: ```shell python examples/stable_diffusion/client.py --prompt "a cute cat playing with a red ball" --output cat.jpg --port 8000 ``` You will get an image named "cat.jpg" in the current directory. You can check the metrics: ```shell curl http://127.0.0.1:8000/metrics ``` That's it! You have just hosted your **_stable-diffusion model_** as a service! 😉

Examples

More ready-to-use examples can be found in the Example section. It includes:

Pipeline: a simple echo demo even without any ML model.
Request validation: validate the request with type annotation and generate OpenAPI documentation.
Multiple route: serve multiple models in one service
Embedding service: OpenAI compatible embedding service
Reranking service: rerank a list of passages based on a query
Shared memory IPC: inter-process communication with shared memory.
Customized GPU allocation: deploy multiple replicas, each using different GPUs.
Customized metrics: record your own metrics for monitoring.
Jax jitted inference: just-in-time compilation speeds up the inference.
Compression: enable request/response compression.
PyTorch deep learning models:
- sentiment analysis: infer the sentiment of a sentence.
- image recognition: categorize a given image.
- stable diffusion: generate images based on texts, with msgpack serialization.

Configuration

Dynamic batching
- max_batch_size and max_wait_time (millisecond) are configured when you call append_worker.
- Make sure inference with the max_batch_size value won't cause the out-of-memory in GPU.
- Normally, max_wait_time should be less than the batch inference time.
- If enabled, it will collect a batch either when the number of accumulated requests reaches max_batch_size or when max_wait_time has elapsed. The service will benefit from this feature when the traffic is high.
Check the arguments doc for other configurations.

Deployment

If you're looking for a GPU base image with mosec installed, you can check the official image mosecorg/mosec. For the complex use case, check out envd.
This service doesn't need Gunicorn or NGINX, but you can certainly use the ingress controller when necessary.
This service should be the PID 1 process in the container since it controls multiple processes. If you need to run multiple processes in one container, you will need a supervisor. You may choose Supervisor or Horust.
Remember to collect the metrics.
- mosec_service_batch_size_bucket shows the batch size distribution.
- mosec_service_batch_duration_second_bucket shows the duration of dynamic batching for each connection in each stage (starts from receiving the first task).
- mosec_service_process_duration_second_bucket shows the duration of processing for each connection in each stage (including the IPC time but excluding the mosec_service_batch_duration_second_bucket).
- mosec_service_remaining_task shows the number of currently processing tasks.
- mosec_service_throughput shows the service throughput.
Stop the service with SIGINT (CTRL+C) or SIGTERM (kill {PID}) since it has the graceful shutdown logic.

Performance tuning

Find out the best max_batch_size and max_wait_time for your inference service. The metrics will show the histograms of the real batch size and batch duration. Those are the key information to adjust these two parameters.
Try to split the whole inference process into separate CPU and GPU stages (ref DistilBERT). Different stages will be run in a data pipeline, which will keep the GPU busy.
You can also adjust the number of workers in each stage. For example, if your pipeline consists of a CPU stage for preprocessing and a GPU stage for model inference, increasing the number of CPU-stage workers can help to produce more data to be batched for model inference at the GPU stage; increasing the GPU-stage workers can fully utilize the GPU memory and computation power. Both ways may contribute to higher GPU utilization, which consequently results in higher service throughput.
For multi-stage services, note that the data passing through different stages will be serialized/deserialized by the serialize_ipc/deserialize_ipc methods, so extremely large data might make the whole pipeline slow. The serialized data is passed to the next stage through rust by default, you could enable shared memory to potentially reduce the latency (ref RedisShmIPCMixin).
You should choose appropriate serialize/deserialize methods, which are used to decode the user request and encode the response. By default, both are using JSON. However, images and embeddings are not well supported by JSON. You can choose msgpack which is faster and binary compatible (ref Stable Diffusion).
Configure the threads for OpenBLAS or MKL. It might not be able to choose the most suitable CPUs used by the current Python process. You can configure it for each worker by using the env (ref custom GPU allocation).
Enable HTTP/2 from client side. mosec automatically adapts to user's protocol (e.g., HTTP/2) since v0.8.8.

Adopters

Here are some of the companies and individual users that are using Mosec:

Modelz: Serverless platform for ML inference.
MOSS: An open sourced conversational language model like ChatGPT.
TencentCloud: Tencent Cloud Machine Learning Platform, using Mosec as the core inference server framework.
TensorChord: Cloud native AI infrastructure company.
OAT: Serving reward models for online LLM alignment.

Citation

If you find this software useful for your research, please consider citing

@software{yang2021mosec, title = {{MOSEC: Model Serving made Efficient in the Cloud}}, author = {Yang, Keming and Liu, Zichen and Cheng, Philip}, howpublished = {https://github.com/mosecorg/mosec}, year = {2021} }

Contributing

We welcome any kind of contribution. Please give us feedback by raising issues or discussing on Discord. You could also directly contribute your code and pull request!

To start develop, you can use envd to create an isolated and clean Python & Rust environment. Check the envd-docs or build.envd for more information.

Owner

Name: mosecorg
Login: mosecorg
Kind: organization
Email: mosecorg@gmail.com

Website: https://mosecorg.github.io
Twitter: mosecorg
Repositories: 1
Profile: https://github.com/mosecorg

Citation (CITATION.cff)

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Yang"
  given-names: "Keming"
  orcid: "https://orcid.org/0000-0002-1351-2342"
- family-names: "Liu"
  given-names: "Zichen"
  orcid: "https://orcid.org/0000-0001-9451-8625"
- family-names: "Cheng"
  given-names: "Philip"
title: "MOSEC: Model Serving made Efficient in the Cloud"
url: "https://github.com/mosecorg/mosec"
type: software
date-released: 2021-09-27

GitHub Events

Total

Create event: 63
Issues event: 4
Release event: 6
Watch event: 79
Delete event: 57
Issue comment event: 58
Push event: 46
Pull request review comment event: 14
Pull request review event: 68
Pull request event: 137
Fork event: 7

Last Year

Create event: 63
Issues event: 4
Release event: 6
Watch event: 79
Delete event: 57
Issue comment event: 58
Push event: 46
Pull request review comment event: 14
Pull request review event: 68
Pull request event: 137
Fork event: 7

Committers

Last synced: 9 months ago

All Time

Total Commits: 417
Total Committers: 14
Avg Commits per committer: 29.786
Development Distribution Score (DDS): 0.489

Past Year

Commits: 56
Committers: 7
Avg Commits per committer: 8.0
Development Distribution Score (DDS): 0.375

Top Committers

Name	Email	Commits
Keming	k**4@g**m	213
dependabot[bot]	4****]	139
zclzc	3****c	46
hang lv	x**0@f**n	7
Ce Gao	c**o@t**i	3
sravan	m**6@g**m	1
delonleo	d****o	1
cutecutecat	s**7@g**m	1
Zhong Qishuai	F**g@g**m	1
Jincheng Miao	j**o@i**m	1
Jangwon Park	a**w@g**m	1
Ikko Eltociear Ashimine	e**r@g**m	1
Aniket Shirsat	a**t@g**m	1
Alan Lee	s**m@o**m	1

Committer Domains (Top 20 + Academic)

intel.com: 1 tensorchord.ai: 1 fudan.edu.cn: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 59
Total pull requests: 405
Average time to close issues: 3 months
Average time to close pull requests: 7 days
Total issue authors: 23
Total pull request authors: 13
Average comments per issue: 1.93
Average comments per pull request: 0.58
Merged pull requests: 284
Bot issues: 1
Bot pull requests: 217

Past Year

Issues: 4
Pull requests: 121
Average time to close issues: 21 days
Average time to close pull requests: 4 days
Issue authors: 4
Pull request authors: 8
Average comments per issue: 0.25
Average comments per pull request: 0.56
Merged pull requests: 75
Bot issues: 0
Bot pull requests: 54

View more stats

Top Authors

Issue Authors

kemingy (26)
gaocegege (6)
lkevinzc (4)
aseaday (2)
decadance-dance (2)
secsilm (2)
VoVAllen (2)
cutecutecat (1)
step21 (1)
ynkim0 (1)
piglaker (1)
lfxx (1)
oliverhu (1)
dependabot[bot] (1)
delonleo (1)

Pull Request Authors

dependabot[bot] (266)
kemingy (161)
lkevinzc (14)
n063h (13)
gaocegege (3)
monologg (2)
cutecutecat (2)
miaojinc (2)
Innovatorcloudy (2)
eltociear (2)
sravan1946 (2)
Smoothengineer (2)
Copilot (1)

Top Labels

Issue Labels

enhancement (44) bug (12) python (5) rust (5) level/hard (4) help wanted (3) chore (2) refactoring (2) level/easy (2) level/medium (2) good first issue (2) dependencies (1) github_actions (1) documentation (1)

Pull Request Labels

chore (322) dependencies (266) rust (182) enhancement (56) python (46) github_actions (39) documentation (34) bug_fix (29) refactoring (4)

Packages

Total packages: 3
Total downloads:
- pypi 3,131 last-month
- cargo 48,861 total
Total docker downloads: 7,202

Total dependent packages: 1
(may contain duplicates)
Total dependent repositories: 1
(may contain duplicates)
Total versions: 96
Total maintainers: 5

pypi.org: mosec

Model Serving made Efficient in the Cloud

Documentation: https://mosec.readthedocs.io/
License: Apache-2.0
Latest release: 0.9.5
published 9 months ago

Versions: 50
Dependent Packages: 1
Dependent Repositories: 1
Downloads: 3,082 Last month
Docker Downloads: 7,202

Rankings

Stargazers count: 2.5%

Docker downloads count: 3.9%

Dependent packages count: 4.8%

Downloads: 5.9%

Forks count: 6.0%

Average: 7.4%

Dependent repos count: 21.6%

Maintainers (2)

keming zclzc

Last synced: 6 months ago

pypi.org: mosec-tiinfer

Model Serving made Efficient in the Cloud.

Homepage: https://github.com/mosecorg/mosec
Documentation: https://mosec-tiinfer.readthedocs.io/
License: Apache License 2.0
Latest release: 0.0.7
published almost 3 years ago

Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 49 Last month

Rankings

Stargazers count: 3.8%

Dependent packages count: 6.9%

Forks count: 7.6%

Average: 14.1%

Downloads: 21.8%

Dependent repos count: 30.6%

Maintainers (1)

ti-platform

Last synced: 6 months ago

crates.io: mosec

Model Serving made Efficient in the Cloud.

Documentation: https://docs.rs/mosec/
License: Apache-2.0
Latest release: 0.9.0
published over 1 year ago

Versions: 45
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 48,861 Total

Rankings

Stargazers count: 9.8%

Forks count: 10.9%

Average: 23.0%

Dependent repos count: 29.3%

Downloads: 30.9%

Dependent packages count: 33.8%

Maintainers (2)

kemingy lkevinzc

Last synced: 6 months ago

Dependencies

Cargo.lock cargo

ansi_term 0.12.1
async-channel 1.6.1
atty 0.2.14
autocfg 1.1.0
bitflags 1.3.2
bytes 1.2.1
cache-padded 1.2.0
cfg-if 1.0.0
clap 3.2.16
clap_derive 3.2.15
clap_lex 0.2.2
concurrent-queue 1.2.2
convert_case 0.4.0
derive_more 0.99.17
event-listener 2.5.2
fnv 1.0.7
futures-channel 0.3.21
futures-core 0.3.21
futures-task 0.3.21
futures-util 0.3.21
hashbrown 0.12.1
heck 0.4.0
hermit-abi 0.1.19
http 0.2.8
http-body 0.4.5
httparse 1.7.1
httpdate 1.0.2
hyper 0.14.20
indexmap 1.9.0
itoa 1.0.2
lazy_static 1.4.0
libc 0.2.126
lock_api 0.4.7
log 0.4.17
matchers 0.1.0
memchr 2.5.0
mio 0.8.3
num_cpus 1.13.1
once_cell 1.13.0
os_str_bytes 6.1.0
parking_lot 0.12.1
parking_lot_core 0.9.3
pin-project-lite 0.2.9
pin-utils 0.1.0
proc-macro-error 1.0.4
proc-macro-error-attr 1.0.4
proc-macro2 1.0.39
prometheus 0.13.1
protobuf 2.27.1
quote 1.0.18
redox_syscall 0.2.13
regex 1.5.6
regex-automata 0.1.10
regex-syntax 0.6.26
rustc_version 0.4.0
scopeguard 1.1.0
semver 1.0.10
sharded-slab 0.1.4
signal-hook-registry 1.4.0
smallvec 1.8.0
socket2 0.4.4
strsim 0.10.0
syn 1.0.96
termcolor 1.1.3
textwrap 0.15.0
thiserror 1.0.31
thiserror-impl 1.0.31
thread_local 1.1.4
tokio 1.20.1
tokio-macros 1.8.0
tower-service 0.3.2
tracing 0.1.36
tracing-attributes 0.1.22
tracing-core 0.1.29
tracing-log 0.1.3
tracing-subscriber 0.3.15
try-lock 0.2.3
unicode-ident 1.0.1
valuable 0.1.0
version_check 0.9.4
want 0.3.0
wasi 0.11.0+wasi-snapshot-preview1
winapi 0.3.9
winapi-i686-pc-windows-gnu 0.4.0
winapi-util 0.1.5
winapi-x86_64-pc-windows-gnu 0.4.0
windows-sys 0.36.1
windows_aarch64_msvc 0.36.1
windows_i686_gnu 0.36.1
windows_i686_msvc 0.36.1
windows_x86_64_gnu 0.36.1
windows_x86_64_msvc 0.36.1

requirements/dev.txt pypi

autoflake >=1.4 development
black >=20.8b1 development
httpx >=0.21.0 development
isort >=5.6 development
msgpack >=1.0.2 development
mypy >=0.910 development
pre-commit >=2.15.0 development
pydocstyle >=6.1.1 development
pylint >=2.13.8 development
pytest >=6 development
pytest-mock >=3.5 development
requests * development
setuptools_scm >=7 development

requirements/doc.txt pypi

mkdocs-gen-files >=0.3.3
mkdocs-material >=7.3.0
mkdocstrings >=0.16.0

.github/actions/cache/action.yml actions

actions/cache v3 composite

.github/workflows/check.yml actions

./.github/actions/cache * composite
actions/checkout v4 composite
actions/setup-node v3 composite
actions/setup-python v4 composite

.github/workflows/codeql.yml actions

actions/checkout v4 composite
github/codeql-action/analyze v2 composite
github/codeql-action/autobuild v2 composite
github/codeql-action/init v2 composite

.github/workflows/label.yml actions

actions/checkout v4 composite

.github/workflows/links.yml actions

actions/checkout v4 composite
lycheeverse/lychee-action v1.8.0 composite

.github/workflows/nightly.yml actions

./.github/actions/cache * composite
actions/checkout v4 composite
actions/setup-python v4 composite

.github/workflows/package.yml actions

./.github/actions/cache * composite
actions/checkout v4 composite
actions/download-artifact v3 composite
actions/setup-python v4 composite
actions/upload-artifact v3 composite
docker/build-push-action v5 composite
docker/login-action v3 composite
docker/setup-buildx-action v3 composite
docker/setup-qemu-action v3 composite

.github/workflows/page.yml actions

./.github/actions/cache * composite
actions/checkout v4 composite
actions/configure-pages v3 composite
actions/deploy-pages v2 composite
actions/setup-python v4 composite
actions/upload-pages-artifact v2 composite

Cargo.toml cargo

.devcontainer/Dockerfile docker

python 3.11 build

Dockerfile docker

${base} latest build

examples/monitor/docker-compose.yml docker

grafana/grafana 8.2.2
prom/prometheus v2.30.0

pyproject.toml pypi

requirements/mixin.txt pypi

msgpack >=1.0.2
msgspec >=0.15.0
numbin >=0.4.0
pyarrow >=0.6.1,<12
redis >=4.0.0

setup.py pypi