Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/microsoft/mlos

MLOS is a project to enable autotuning for systems.

https://github.com/microsoft/mlos

Science Score: 59.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 2 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: arxiv.org, acm.org
  • ✓
    Committers with academic emails
    1 of 16 committers (6.3%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.8%) to scientific vocabulary

Keywords

autotuning benchmarking benchmarking-framework data-science infrastructure optimize-systems performance-engineering

Keywords from Contributors

sequences projection interactive serializer measurement cycles packaging charts network-simulation archival
Last synced: 6 months ago · JSON representation

Repository

MLOS is a project to enable autotuning for systems.

Basic Info
Statistics
  • Stars: 165
  • Watchers: 12
  • Forks: 73
  • Open Issues: 117
  • Releases: 7
Topics
autotuning benchmarking benchmarking-framework data-science infrastructure optimize-systems performance-engineering
Created almost 6 years ago · Last pushed 7 months ago
Metadata Files
Readme Contributing License Code of conduct Codeowners Security

README.md

MLOS

MLOS DevContainer MLOS Linux MLOS MacOS MLOS Windows Code Coverage Status pre-commit.ci status

MLOS is a project to enable autotuning for systems.

Contents

Overview

MLOS currently focuses on an offline tuning approach, though we intend to add online tuning in the future.

To accomplish this, the general flow involves

  • Running a workload (i.e., benchmark) against a system (e.g., a database, web server, or key-value store).
  • Retrieving the results of that benchmark, and perhaps some other metrics from the system.
  • Feed that data to an optimizer (e.g., using Bayesian Optimization or other techniques).
  • Obtain a new suggested config to try from the optimizer.
  • Apply that configuration to the target system.
  • Repeat until either the exploration budget is consumed or the configurations' performance appear to have converged.

optimization loop

Source: LlamaTune: VLDB 2022

For a brief overview of some of the features and capabilities of MLOS, please see the following video:

demo video

Organization

To do this this repo provides three Python modules, which can be used independently or in combination:

  • mlos-bench provides a framework to help automate running benchmarks as described above.

  • mlos-viz provides some simple APIs to help automate visualizing the results of benchmark experiments and their trials.

It provides a simple plot(experiment_data) API, where experiment_data is obtained from the mlos_bench.storage module.

  • mlos-core provides an abstraction around existing optimization frameworks (e.g., FLAML, SMAC, etc.)

It is intended to provide a simple, easy to consume (e.g. via pip), with low dependencies abstraction to

  • describe a space of context, parameters, their ranges, constraints, etc. and result objectives
  • an "optimizer" service abstraction (e.g. register() and suggest()) so we can easily swap out different implementations methods of searching (e.g. random, BO, LLM, etc.)
  • provide some helpers for automating optimization experiment runner loops and data collection

For these design requirements we intend to reuse as much from existing OSS libraries as possible and layer policies and optimizations specifically geared towards autotuning systems over top.

By providing wrappers we aim to also allow more easily experimenting with replacing underlying optimizer components as new techniques become available or seem to be a better match for certain systems.

Contributing

See CONTRIBUTING.md for details on development environment and contributing.

Getting Started

The development environment for MLOS uses conda and devcontainers to ease dependency management, but not all these libraries are required for deployment.

For instructions on setting up the development environment please try one of the following options:

  • see CONTRIBUTING.md for details on setting up a local development environment
  • launch this repository (or your fork) in a codespace, or
  • have a look at one of the autotuning example repositories like sqlite-autotuning to kick the tires in a codespace in your browser immediately :)

conda activation

  1. Create the mlos Conda environment.

sh conda env create -f conda-envs/mlos.yml

See the conda-envs/ directory for additional conda environment files, including those used for Windows (e.g. mlos-windows.yml).

or

sh # This will also ensure the environment is update to date using "conda env update -f conda-envs/mlos.yml" make conda-env

Note: the latter expects a *nix environment.

  1. Initialize the shell environment.

sh conda activate mlos

Usage Examples

mlos-core

For an example of using the mlos_core optimizer APIs run the BayesianOptimization.ipynb notebook.

mlos-bench

For an example of using the mlos_bench tool to run an experiment, see the mlos_bench Quickstart README.

Here's a quick summary:

```shell ./scripts/generate-azure-credentials-config > globalconfigazure.jsonc

run a simple experiment

mlosbench --config ./mlosbench/mlos_bench/config/cli/azure-redis-1shot.jsonc ```

See Also:

mlos-viz

For a simple example of using the mlos_viz module to visualize the results of an experiment, see the sqlite-autotuning repository, especially the mlosdemosqlite_teachers.ipynb notebook.

Installation

The MLOS modules are published to pypi when new releases are tagged:

To install the latest release, simply run:

```sh

this will install just the optimizer component with SMAC support:

pip install -U mlos-core[smac]

this will install just the optimizer component with flaml support:

pip install -U "mlos-core[flaml]"

this will install just the optimizer component with smac and flaml support:

pip install -U "mlos-core[smac,flaml]"

this will install both the flaml optimizer and the experiment runner with azure support:

pip install -U "mlos-bench[flaml,azure]"

this will install both the smac optimizer and the experiment runner with ssh support:

pip install -U "mlos-bench[smac,ssh]"

this will install the postgres storage backend for mlos-bench

and mlos-viz for visualizing results:

pip install -U "mlos-bench[postgres]" mlos-viz ```

Details on using a local version from git are available in CONTRIBUTING.md.

See Also

Examples

Working example of tuning sqlite with MLOS.

These can be used as starting points for new autotuning projects outside of the main MLOS repository if you want to keep your tuning experiment configs separate from the MLOS codebase.

Alternatively, we accept PRs to add new examples to the main MLOS repository! See mlos_bench/config and CONTRIBUTING.md for more details.

Publications

Owner

  • Name: Microsoft
  • Login: microsoft
  • Kind: organization
  • Email: opensource@microsoft.com
  • Location: Redmond, WA

Open source projects and samples from Microsoft

GitHub Events

Total
  • Create event: 5
  • Release event: 1
  • Issues event: 45
  • Watch event: 24
  • Delete event: 5
  • Member event: 2
  • Issue comment event: 89
  • Push event: 324
  • Pull request review comment event: 290
  • Pull request review event: 363
  • Pull request event: 159
  • Fork event: 11
Last Year
  • Create event: 5
  • Release event: 1
  • Issues event: 45
  • Watch event: 24
  • Delete event: 5
  • Member event: 2
  • Issue comment event: 89
  • Push event: 324
  • Pull request review comment event: 290
  • Pull request review event: 363
  • Pull request event: 159
  • Fork event: 11

Committers

Last synced: 9 months ago

All Time
  • Total Commits: 545
  • Total Committers: 16
  • Avg Commits per committer: 34.063
  • Development Distribution Score (DDS): 0.484
Past Year
  • Commits: 130
  • Committers: 6
  • Avg Commits per committer: 21.667
  • Development Distribution Score (DDS): 0.238
Top Committers
Name Email Commits
Brian Kroth b****h 281
Sergiy Matusevych s****m@m****m 178
Brian Kroth B****h@m****m 39
Eu Jing Chua c****g@g****m 13
Andreas Mueller a****r@m****m 7
dependabot[bot] 4****] 5
Eu Jing Chua e****a@m****m 5
Johannes Freischuetz z****2@g****m 4
Konstantinos Kanellis k****s@c****u 3
Jacqueline Maureen j****n@m****m 2
Konstantinos Kanellis v****s@m****m 2
Omisa Jinsi o****i@m****m 2
Kelly Kostopoulou D****y 1
anjagruenheid 8****d 1
poojanilangekar n****a@g****m 1
ndhuynh h****y@g****m 1
Committer Domains (Top 20 + Academic)
microsoft.com: 7 cs.wisc.edu: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 36
  • Total pull requests: 158
  • Average time to close issues: about 2 months
  • Average time to close pull requests: 16 days
  • Total issue authors: 4
  • Total pull request authors: 8
  • Average comments per issue: 1.33
  • Average comments per pull request: 0.54
  • Merged pull requests: 113
  • Bot issues: 0
  • Bot pull requests: 8
Past Year
  • Issues: 29
  • Pull requests: 151
  • Average time to close issues: 5 days
  • Average time to close pull requests: 6 days
  • Issue authors: 4
  • Pull request authors: 7
  • Average comments per issue: 0.62
  • Average comments per pull request: 0.46
  • Merged pull requests: 110
  • Bot issues: 0
  • Bot pull requests: 6
View more stats
Top Authors
Issue Authors
  • bpkroth (69)
  • motus (15)
  • eujing (4)
  • yshady (3)
  • ephoris (1)
  • DelphianCalamity (1)
  • jsfreischuetz (1)
  • yshady-acheev (1)
Pull Request Authors
  • bpkroth (232)
  • motus (56)
  • dependabot[bot] (16)
  • jsfreischuetz (14)
  • eujing (14)
  • yshady (7)
  • yshady-acheev (6)
  • DelphianCalamity (3)
  • kkanellis (2)
  • ephoris (1)
Top Labels
Issue Labels
mlos-bench (8) enhancement (7) bug (7) tests (6) dependencies (3) low priority (3) mlos-core (3) ci (3) documentation (2) WIP (1) design (1) github_actions (1) mlos-viz (1) python (1) dev-env (1)
Pull Request Labels
ready for review (163) mlos-bench (38) bug (33) dependencies (30) tests (24) ci (23) documentation (17) python (15) WIP (14) refactor (14) mlos-core (9) low priority (9) enhancement (7) dev-env (5) mlos-viz (3) github_actions (2) docker (1)

Packages

  • Total packages: 5
  • Total downloads:
    • pypi 198 last-month
  • Total dependent packages: 2
    (may contain duplicates)
  • Total dependent repositories: 0
    (may contain duplicates)
  • Total versions: 56
  • Total maintainers: 1
proxy.golang.org: github.com/microsoft/MLOS
  • Versions: 16
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 6.4%
Average: 6.6%
Dependent repos count: 6.8%
Last synced: 7 months ago
proxy.golang.org: github.com/microsoft/mlos
  • Versions: 16
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 6.4%
Average: 6.6%
Dependent repos count: 6.8%
Last synced: 7 months ago
pypi.org: mlos-core

MLOS Core Python interface for parameter optimization.

  • Versions: 9
  • Dependent Packages: 1
  • Dependent Repositories: 0
  • Downloads: 72 Last month
Rankings
Dependent packages count: 10.0%
Downloads: 21.4%
Average: 32.6%
Dependent repos count: 66.3%
Maintainers (1)
mlos-maintainers
Last synced: 7 months ago
pypi.org: mlos-bench

MLOS Bench Python interface for benchmark automation and optimization.

  • Versions: 9
  • Dependent Packages: 1
  • Dependent Repositories: 0
  • Downloads: 72 Last month
Rankings
Dependent packages count: 10.0%
Downloads: 23.0%
Average: 33.1%
Dependent repos count: 66.3%
Maintainers (1)
mlos-maintainers
Last synced: 7 months ago
pypi.org: mlos-viz

Visualization Python interface for benchmark automation and optimization results.

  • Versions: 6
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 54 Last month
Rankings
Dependent packages count: 10.0%
Average: 38.0%
Dependent repos count: 66.0%
Maintainers (1)
mlos-maintainers
Last synced: 7 months ago

Dependencies

.github/workflows/codeql.yml actions
  • actions/checkout v4 composite
  • github/codeql-action/analyze v2 composite
  • github/codeql-action/autobuild v2 composite
  • github/codeql-action/init v2 composite
.github/workflows/devcontainer.yml actions
  • JamesIves/github-pages-deploy-action v4 composite
  • actions/checkout v4 composite
  • docker/login-action v3 composite
.github/workflows/linux.yml actions
  • actions/cache v3 composite
  • actions/checkout v4 composite
  • conda-incubator/setup-miniconda v2 composite
.github/workflows/markdown-link-check.yml actions
  • actions/checkout v4 composite
  • gaurav-nelson/github-action-markdown-link-check v1 composite
.devcontainer/Dockerfile docker
  • base latest build
  • mcr.microsoft.com/devcontainers/miniconda 3 build
.devcontainer/build/Dockerfile docker
  • node lts-slim build
.devcontainer/docker-compose.yml docker
doc/Dockerfile docker
  • nginx latest build
doc/requirements.txt pypi
  • doc8 *
  • jupyter_core >=4.11.2
  • mistune >=2.0.3
  • nbconvert *
  • nbsphinx *
  • numpydoc *
  • rstcheck *
  • sphinx *
  • sphinx-rtd-theme *
.github/workflows/windows.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
mlos_bench/setup.py pypi
  • importlib_resources *
  • json5 *
  • jsonschema >=4.18.0
  • mlos-core ==
  • referencing >=0.29.1
  • requests *
mlos_core/setup.py pypi
  • ConfigSpace >=0.7.1
  • joblib >=1.1.1
  • numpy >=1.24
  • pandas >=1.0.3
  • scikit-learn >=1.2
  • scipy >=1.3.2
mlos_bench/mlos_bench/tests/services/remote/ssh/Dockerfile docker
  • debian latest build
mlos_bench/mlos_bench/tests/services/remote/ssh/docker-compose.yml docker
  • mlos_bench-test-ssh-server latest
conda-envs/environment.yml conda
  • flake8
  • ipykernel
  • jupyter
  • libpq
  • matplotlib
  • nb_conda_kernels
  • pip
  • pycodestyle
  • pydocstyle
  • pylint
  • python
  • seaborn
  • setuptools
  • setuptools-scm
  • swig