Recent Releases of https://github.com/awslabs/multi-model-server
https://github.com/awslabs/multi-model-server - v1.1.11 - Extra error logging and new timeout config
What's Changed
- Log model loading exception to assist with debugging by @namannandan in https://github.com/awslabs/multi-model-server/pull/1011
- Add config for request timeout in seconds by @davidthomas426 in https://github.com/awslabs/multi-model-server/pull/1019
- Update version.py by @davidthomas426 in https://github.com/awslabs/multi-model-server/pull/1021
New Contributors
- @namannandan made their first contribution in https://github.com/awslabs/multi-model-server/pull/1011
- @davidthomas426 made their first contribution in https://github.com/awslabs/multi-model-server/pull/1019
Full Changelog: https://github.com/awslabs/multi-model-server/compare/v1.1.10...v1.1.11
- Java
Published by davidthomas426 almost 3 years ago
https://github.com/awslabs/multi-model-server - v1.1.10 - Upgrade gson to 2.8.9
Upgrade gson to 2.8.9 Fixes CVE-2022-25647
- Java
Published by maaquib about 3 years ago
https://github.com/awslabs/multi-model-server - v1.1.9 - Upgrade netty to 4.1.91.Final
Upgrade netty to 4.1.91.Final
- Java
Published by maaquib about 3 years ago
https://github.com/awslabs/multi-model-server - v1.1.8 - Upgrade log4j to 2.17.1
Upgrade log4j to 2.17.1 to fixe CVE-2021-44832 in 2.17.0
- Java
Published by maaquib over 4 years ago
https://github.com/awslabs/multi-model-server - v1.1.7 - Upgrade log4j to 2.17.0
Upgrade log4j2 version to 2.17.0 to address CVE-2021-45046 and CVE-2021-45105
- Java
Published by maaquib over 4 years ago
https://github.com/awslabs/multi-model-server - v1.1.6 - Added disruptor dependency for async logging
- Added disruptor dependency for async logging https://github.com/awslabs/multi-model-server/pull/984
- If using
sagemaker-inference-toolkit, upgrade to version >= v1.5.9
- Java
Published by maaquib over 4 years ago
https://github.com/awslabs/multi-model-server - v1.1.5 - Upgrade log4j to 2.16.0
- Upgrade log4j2 version to 2.16.0 to address CVE-2021-44228 and CVE-2021-45046
- Updated logging docs to address migration from log4j v1 to v2
- Code fix for setting
preload_modeldefault as null for register model request - Fix channel closures in ModelServerTest
- Java
Published by maaquib over 4 years ago
https://github.com/awslabs/multi-model-server - v1.1.4- Fix Bug in Support for custom error codes
- Java
Published by lxning about 5 years ago
https://github.com/awslabs/multi-model-server - v1.1.3 - Support for custom error codes
Allows custom HTTP status in mms.service.Service to be returned to client
- Java
Published by maaquib about 5 years ago
https://github.com/awslabs/multi-model-server - v1.1.2 - Dependency updates for Netty
This release contains minor fixes to bump up Netty & Log4j versions.
- Java
Published by dhanainme almost 6 years ago
https://github.com/awslabs/multi-model-server - v1.1.1 - Resource cleanup for terminated worker threads
This release contains minor fixes to make sure resource cleaning is done for terminated worker threads * Terminates the STDOUT and STDERR ReaderThreads for a Worker when it is scaled down
- Java
Published by maaquib about 6 years ago
https://github.com/awslabs/multi-model-server - v1.0.8 - Synchronous resource cleaning and API changes
This release contains API changes and fixes to make sure resource cleaning is handled synchronously. * Load model API sends a conflict response instead of a bad request response when trying to register an already registered model. #851 * Unregister model API is now synchronous and will wait until all resources are cleaned before sending a response back. A timeout feature was also added to config if users don't want to wait. #853
- Java
Published by alexwong over 6 years ago
https://github.com/awslabs/multi-model-server - v1.0.7 - Bug fixes to support python 2 better
This release contains a minor bug fix for Python 2 support. * Changed the python protocol handler between frontend and backend to support python 2 better.
- Java
Published by vdantu almost 7 years ago
https://github.com/awslabs/multi-model-server - v1.0.6 - Features to handle OOM errors and enhancements to configurability of MMS
- Load model API takes in JSON requests. #818
- Implementation of Ping API using the plugins SDK. #814
- Newer endpoint for predictions.
POST /models/{model-id}/invoke. #823 - Handling OOM errors. MMS returns a HTTP 507 error code when there is a OOM error during runtime of MMS. #822
- Added changes to allow MMS have the same Management and Inference addresses #826
- Changes to MMS default behavior. MMS by default runs
POST /modelsin a synchronous way and if there aredefault_workers_per_model, this value will be used when loading models. #836 - MMS configuration values can take environment variables. #841
- Java
Published by vdantu almost 7 years ago
https://github.com/awslabs/multi-model-server - v1.0.5 - Model Server support for plugins
This release contains multiple model server changes
Major features
- Plugins support
- SDK for plugins
- Reference plugins implementation
- MMS changes to support plugins
- Feature to support default service file configured.
- Feature to support return of custom HTTP headers from the model.
Minor features
- Option to run MMS in the foreground .... And multiple bug fixes
- Java
Published by vdantu almost 7 years ago
https://github.com/awslabs/multi-model-server - v1.0.4 - Contains model-archiver features, integration test framework and bug fixes
This release contains multiple model-archiver features and bug fixes.
Features
- Added support for "no-archive"
- Added feature to support optional conversion of ONNX model to MXNet model
- Added integration test framework for model-archiver.
- Java
Published by vdantu about 7 years ago
https://github.com/awslabs/multi-model-server - v1.0.3 - Base MMS containers available
Features and Bug Fixes
- Published base MMS containers for Python 2.7 and Python 3.6 with Ubuntu 16.04 and nvidia/cuda 9.2 with CUDNN 7 on ubuntu 16.04.
- model-archiver changes to handle multiple archive formats
- model-server configurable through environment variables
- Contains multiple bug fixes
- Java
Published by vdantu about 7 years ago
https://github.com/awslabs/multi-model-server - v1.0.2 - Multiple features and bug fixes
In this release we have addressed all the reported bugs and also added enhancements such as
Features and Major Bug Fixes
- Frontend listening on Unix Domain Socket.
- Support Asynchronous logging.
- Added documentation for batching support.
- Added features to support
- Starting default number of workers for models that are launched at MMS Startup time.
- Configurable response time out for individual models. This is the amount of time MMS waits for the model to respond to a request.
- Configure Maximum allowable request and response sizes.
- Changes for new Container images.
- Passing all HTTP headers to the backend worker.
- Adding shufflenet to the model server model-zoo.
- Adding example to bring sockeye model onto MMS. ... And bug fixes
- Java
Published by vdantu over 7 years ago
https://github.com/awslabs/multi-model-server - v1.0.1 - Apache Model Server for MXNet adds minor features and addresses bugs
In this release of MXNet Model Server, we have added the following features.
Features and Bug fixes
- Changes for batching support.
- CORS headers support added to responses.
- Handle content-type returned by the backend code and pass ContentType to the service code
- Workaround import mxnet module timeout issue. Now MXNet startup time doesn't cause significant delay upon MMS start on compute optimized hosts
- Make sure that python prints are not buffered
- Refactor metrics emission logic
- Always use utf-8 to decode bytes.
- Avoid archiving a model archive file recursively.
- Pythonpath issues for MMS
- Documentation updates
- Java
Published by vdantu over 7 years ago
https://github.com/awslabs/multi-model-server - Apache Model Server for MXNet adds support for hot loading of models
In this release of MXNet Model Server, we have added the following major features.
Features
- Loading and Unloading models at run-time (hot loading models). This is now available via management REST API exposed by MMS. More on management API here
- Independently scale number of model-worker instances serving inference requests. This is available through management REST API.
- Improved model archive representation. More on model-archiver is here
- Improved docker container images.
- Improved performance compared to MMS v0.4 and decreased dependencies. One of the major changes is replacing monolithic architecture with separate frontend and backend. Netty is used as frontend webserver instead of Flask+GUnicorn combo. Python is for the backend.
- Improved logging and metrics collection. Using log4j and corresponding config to control metrics, including custom user metrics. More on logging config is here
New and updated documents:
- Migration document to migrate from MMS 0.4 to MMS 1.0.
- New Management API.
- Updated model zoo.
- Updated Inference API.
For further documentation, please refer /docs folder
Bug fixes:
This release fixes all the bugs logged on GitHub.
- Java
Published by vdantu over 7 years ago
https://github.com/awslabs/multi-model-server - v0.4.0 - Adding support for Gluon imperative models
New Features:
Gluon imperative model support
- Added support for serving Gluon based imperative models.
Docs, Improvements, Bug fixes
Docs: * Added documentation on Gluon model services. * Added alexnet in Gluon to model serving to examples. * Added Character-level Convolutional Neural Networks in Gluon to model serving examples.
Improvements: * Gluon base service class implementation. (@vdantu ) * Improved Docker image setup time by , layering docker images. (@vrakesh, #343 ) * Docker image now can auto-detect number of available CPUs. (@vrakesh, #360 ) * Added pylint support. (@vdantu) * Use cu90mkl mxnet on cuda gpu machines by default. (@vrakesh, #390 )
Bug Fixes: * Fixed an issue, where empty folder was created when invalid model path is specified. (@vrakesh, #320 ) * Docker images now do not allow multiple instances of MMS to run. (@vrakesh, #337 ) * fixed pypi summary issue. (@aaronmarkham, #378 ) * Fixed error propagation from custom service to MMS. (@vrakesh, #387 ) * Fixed documentation bugs. (@vrakesh, #401 , #402 ) * Fixed version reading issue in MMS. (@vrakesh, #395 ) * Fixed post process latencies being high due to inference variables being lazy evaluated. (@vrakesh, #414 )
- Java
Published by lupesko about 8 years ago
https://github.com/awslabs/multi-model-server - v0.3.0
New Features:
New CLI to interact with MMS running in a container
- New options to start/stop/restart MMS in container.
- Option to point to different configuration files for each MMS run.
- Multiple bug fixes.
Optimized and pre-configured MMS container images
- Published the container image to Docker Hub.
- The default configuration in these containers and the example configuration in the repository are optimized for CPU and GPU AWS EC2 instances.
Bug fixes and Docs
Docs: * README documents. * Added docs to depict orchestrating MMS as an AWS FARGATE service. * Added docs for optimizing the MMS configuration for different EC2 instances.
Bug Fixes: * Corrected Readme and advanced-settings doc for MMS container (@aaronmarkham ) * Documentation for optimised setup for GPU and CPU EC2 instances (@ankkhedia ) * Optimized MMS GPU-container to utilize all GPUs in an GPU instance (@ankkhedia ) * Documentation for launching MMS on AWS Fargate service * Added integration tests framework (@ankkhedia ) * Doc update on Production usage. Describes why Container images are better for prod. (#336) * Streamlining Container based MMS orchestration (@vdantu) * Optimized the model file downloads for container runs of MMS. (@vdantu) * Fixed bugs in mxnet-model-export (@ankkhedia )
- Java
Published by lupesko about 8 years ago
https://github.com/awslabs/multi-model-server - v0.2.0
New features
ONNX model support
Model server now supports models stored in the Open Neural Network Exchange (ONNX) format. See Export an ONNX Model for details.
Cloudwatch metrics
Model server can publish host and model related metrics to Amazon Cloudwatch. See Cloudwatch metrics for details.
Improvements and bug fixes
- Fixing LatencyOverall unit reporting (@lupesko, #317)
- update onnx-mxnet (@jesterhazy, #316)
- remove docs images (@jesterhazy, #315)
- added cloudwatch metrics section (@aaronmarkham, #314)
- update docker scripts (@jesterhazy, #313)
- added toc, logos, and kitten image (@aaronmarkham, #311)
- add unit test for hyphenated model files (@jesterhazy, #308)
- Fix validateprefixmatch (@knjcode, #307)
- align metrics names/units with standard cloudwatch metrics (@jesterhazy, #303)
- Fix race condition when multiple gunicorn workers try to download same models (@yuruofeifei, #302)
- Fix epoch number validation (@knjcode, #298)
- remove License field (@jesterhazy, #297)
- update error messages for model export (@jesterhazy, #295)
- Fixing and updating docker setup (@lupesko, #294)
- public domain image examples for SSD outputs (@aaronmarkham, #291)
- refactored export info; added shortcuts and other improvements (@aaronmarkham, #290)
- added four onnx-exported models to zoo; added onnx support to server intro (@aaronmarkham, #288)
- Documentation updates for 0.2 (@aaronmarkham, #285)
- Improve cloudwatch integration, fix several issues. (@yuruofeifei, #283)
- fail fast when user tries to serve onnx model directly (@jesterhazy, #280)
- fix importlib warning (#254) (@jesterhazy, #279)
- bump version (@yuruofeifei, #250)
- Onnx and metrics docs (@yuruofeifei, #244)
- Add onnx support (@yuruofeifei, #240)
- Zoo updates (@aaronmarkham, #234)
- Zoo updates with details for each model (@aaronmarkham, #233)
- Java
Published by jesterhazy over 8 years ago
https://github.com/awslabs/multi-model-server - Initial release of Model Server
Key capabilities of Model Server for Apache MXNet v0.1.5: - Tooling to package and export all model artifacts into a single “model archive” file that encapsulates everything required for serving an MXNet model. - Automated setup of a serving stack, including HTTP inference endpoints, MXNet-based engine, all automatically configured for the specific models being hosted. - Pre-configured Docker images, setup with NGINX, MXNet and MMS, for scalable model serving. - Ability to customize every step in the inference execution pipeline, from model initialization, through pre-processing and inference, and up to post-processing the model’s output. - Real time operational metrics to monitor the inference service and endpoints, covering key metrics such as latencies, resource utilization and errors. - OpenAPI-enabled service, that is easy to integrate with, and that can auto-generate client code for popular stacks such as Java, JavaScript, C# and more.
- Java
Published by lupesko over 8 years ago