https://github.com/netflix/metaflow
Build, Manage and Deploy AI/ML Systems
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
3 of 105 committers (2.9%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (13.1%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
Build, Manage and Deploy AI/ML Systems
Basic Info
- Host: GitHub
- Owner: Netflix
- License: apache-2.0
- Language: Python
- Default Branch: master
- Homepage: https://metaflow.org
- Size: 44.3 MB
Statistics
- Stars: 9,444
- Watchers: 292
- Forks: 874
- Open Issues: 356
- Releases: 0
Topics
Metadata Files
README.md
Metaflow
Metaflow is a human-centric framework designed to help scientists and engineers build and manage real-life AI and ML systems. Serving teams of all sizes and scale, Metaflow streamlines the entire development lifecycle—from rapid prototyping in notebooks to reliable, maintainable production deployments—enabling teams to iterate quickly and deliver robust systems efficiently.
Originally developed at Netflix and now supported by Outerbounds, Metaflow is designed to boost the productivity for research and engineering teams working on a wide variety of projects, from classical statistics to state-of-the-art deep learning and foundation models. By unifying code, data, and compute at every stage, Metaflow ensures seamless, end-to-end management of real-world AI and ML systems.
Today, Metaflow powers thousands of AI and ML experiences across a diverse array of companies, large and small, including Amazon, Doordash, Dyson, Goldman Sachs, Ramp, and many others. At Netflix alone, Metaflow supports over 3000 AI and ML projects, executes hundreds of millions of data-intensive high-performance compute jobs processing petabytes of data and manages tens of petabytes of models and artifacts for hundreds of users across its AI, ML, data science, and engineering teams.
From prototype to production (and back)
Metaflow provides a simple and friendly pythonic API that covers foundational needs of AI and ML systems:
- Rapid local prototyping, support for notebooks, and built-in support for experiment tracking, versioning and visualization.
- Effortlessly scale horizontally and vertically in your cloud, utilizing both CPUs and GPUs, with fast data access for running massive embarrassingly parallel as well as gang-scheduled compute workloads reliably and efficiently.
- Easily manage dependencies and deploy with one-click to highly available production orchestrators with built in support for reactive orchestration.
For full documentation, check out our API Reference or see our Release Notes for the latest features and improvements.
Getting started
Getting up and running is easy. If you don't know where to start, Metaflow sandbox will have you running and exploring in seconds.
Installing Metaflow
To install Metaflow in your Python environment from PyPI:
sh
pip install metaflow
Alternatively, using conda-forge:
sh
conda install -c conda-forge metaflow
Once installed, a great way to get started is by following our tutorial. It walks you through creating and running your first Metaflow flow step by step.
For more details on Metaflow’s features and best practices, check out:
- How Metaflow works
- Additional resources
If you need help, don’t hesitate to reach out on our Slack community!
Deploying infrastructure for Metaflow in your cloud
While you can get started with Metaflow easily on your laptop, the main benefits of Metaflow lie in its ability to scale out to external compute clusters and to deploy to production-grade workflow orchestrators. To benefit from these features, follow this guide to configure Metaflow and the infrastructure behind it appropriately.
Get in touch
We'd love to hear from you. Join our community Slack workspace!
Contributing
We welcome contributions to Metaflow. Please see our contribution guide for more details.
Owner
- Name: Netflix, Inc.
- Login: Netflix
- Kind: organization
- Email: netflixoss@netflix.com
- Location: Los Gatos, California
- Website: http://netflix.github.io/
- Repositories: 228
- Profile: https://github.com/Netflix
Netflix Open Source Platform
Committers
Last synced: 10 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Savin | s****l@g****m | 316 |
| Romain | r****l | 240 |
| Sakari Ikonen | 6****n | 186 |
| Valay Dave | v****g@g****m | 69 |
| madhur-ob | 1****b | 63 |
| Oleg Avdeev | o****v@g****m | 61 |
| Aapo Kyrola | a****a@c****u | 29 |
| dependabot[bot] | 4****] | 23 |
| Jason Ge | j****e@n****m | 19 |
| jackie-ob | 1****b | 15 |
| Darin | d****7@g****m | 14 |
| Shashank Srikanth | s****h@n****m | 14 |
| David Neuzerling | m****g | 10 |
| Saurabh Garg | 1****b | 9 |
| Chaoying Wang | c****w@n****m | 9 |
| Ville Tuulos | t****s@g****m | 9 |
| Christian Clauss | c****s@m****m | 8 |
| Adam Merberg | a****g@g****m | 7 |
| Shri Javadekar | s****j | 7 |
| David Poznik | d****k | 7 |
| Preetam Joshi | p****3@g****m | 6 |
| Tom Furmston | t****n@g****m | 6 |
| Eddie Mattia | 4****a | 4 |
| Nissan Pow | n****w | 4 |
| ferras | f****7@g****m | 3 |
| bishax | a****p@n****k | 3 |
| Tyler | 4****s | 3 |
| Sri Datta Budaraju | b****a@g****m | 3 |
| Brendan Gibson | 9****n | 3 |
| rohanrebello | 1****o | 2 |
| and 75 more... | ||
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 323
- Total pull requests: 1,597
- Average time to close issues: about 1 year
- Average time to close pull requests: 28 days
- Total issue authors: 179
- Total pull request authors: 120
- Average comments per issue: 2.15
- Average comments per pull request: 0.69
- Merged pull requests: 1,079
- Bot issues: 0
- Bot pull requests: 29
Past Year
- Issues: 73
- Pull requests: 779
- Average time to close issues: 15 days
- Average time to close pull requests: 7 days
- Issue authors: 48
- Pull request authors: 53
- Average comments per issue: 0.85
- Average comments per pull request: 0.49
- Merged pull requests: 522
- Bot issues: 0
- Bot pull requests: 18
Top Authors
Issue Authors
- savingoyal (29)
- tuulos (20)
- saikonen (16)
- romain-intel (13)
- wrighting (6)
- tylerpotts (5)
- dennismoe (5)
- oavdeev (4)
- dhpollack (4)
- svpino (4)
- shrinandj (4)
- ShantanuKumar (3)
- milesgranger (3)
- yank666 (3)
- sangwoo-joh (3)
Pull Request Authors
- saikonen (380)
- savingoyal (252)
- madhur-ob (191)
- romain-intel (171)
- valayDave (123)
- talsperre (61)
- darinyu (36)
- dependabot[bot] (29)
- iamsgarg-ob (25)
- wangchy27 (23)
- npow (19)
- shrinandj (15)
- oavdeev (13)
- amerberg (11)
- trhodeos (10)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 4
-
Total downloads:
- pypi 405,007 last-month
- Total docker downloads: 17,174,130
-
Total dependent packages: 23
(may contain duplicates) -
Total dependent repositories: 107
(may contain duplicates) - Total versions: 390
- Total maintainers: 3
pypi.org: metaflow
Metaflow: More AI and ML, Less Engineering
- Documentation: https://docs.metaflow.org
- License: Apache Software License
-
Latest release: 2.18.2
published 6 months ago
Rankings
Maintainers (2)
conda-forge.org: metaflow
- Homepage: https://metaflow.org/
- License: Apache-2.0
-
Latest release: 2.7.14
published over 3 years ago
Rankings
pypi.org: metaflow-suanpan
Metaflow with suanpan plugin, based on metaflow 2.12.22
- Documentation: https://docs.metaflow.org
- License: Apache Software License
-
Latest release: 0.0.6
published over 1 year ago
Rankings
Maintainers (1)
pypi.org: metaflow-stubs
Metaflow Stubs: Stubs for the metaflow package
- Documentation: https://docs.metaflow.org
- License: Apache Software License
-
Latest release: 2.18.2
published 6 months ago
Rankings
Maintainers (1)
Dependencies
- actions/checkout v3 composite
- github/codeql-action/analyze v2 composite
- github/codeql-action/autobuild v2 composite
- github/codeql-action/init v2 composite
- actions/checkout ee0669bd1cc54295c223e0bb666b733df41de1c5 composite
- actions/setup-python 75f3110429a8c05be0e1bf360334e4cced2b63fa composite
- actions/checkout ee0669bd1cc54295c223e0bb666b733df41de1c5 composite
- actions/setup-python 152ba7c4dd6521b8e9c93f72d362ce03bf6c4f20 composite
- pypa/gh-action-pypi-publish 9b8e7336db3f96a2939a3e9fa827c62f466ca60d composite
- actions/checkout ee0669bd1cc54295c223e0bb666b733df41de1c5 composite
- actions/setup-python 75f3110429a8c05be0e1bf360334e4cced2b63fa composite
- pre-commit/action 9b88afc9cd57fd75b655d5c71bd38146d07135fe composite
- r-lib/actions/setup-r 33f03a860e4659235eb60a4d87ebc0b2ea65f722 composite
- R6 * imports
- digest >= 0.4.0 imports
- magrittr * imports
- reticulate >= 1.10 imports
- cli * suggests
- knitr * suggests
- lubridate * suggests
- rmarkdown * suggests
- testthat * suggests
- @rollup/plugin-commonjs ^21.0.1 development
- @rollup/plugin-node-resolve ^13.1.3 development
- @rollup/plugin-typescript ^8.3.0 development
- @tsconfig/svelte ^3.0.0 development
- @types/node ^17.0.8 development
- @typescript-eslint/eslint-plugin ^5.61.0 development
- @typescript-eslint/parser ^5.61.0 development
- cross-env ^7.0.3 development
- cypress 9.2.1 development
- cypress-svelte-unit-test ^3.3.4 development
- eslint ^8.44.0 development
- eslint-plugin-svelte3 ^3.4.0 development
- postcss ^8.4.24 development
- rollup ^2.63.0 development
- rollup-plugin-css-only ^3.1.0 development
- rollup-plugin-livereload ^2.0.5 development
- rollup-plugin-postcss ^4.0.2 development
- rollup-plugin-svelte ^7.1.6 development
- rollup-plugin-terser ^7.0.2 development
- sirv-cli ^2.0.1 development
- svelte ^3.49.0 development
- svelte-check ^2.3.0 development
- svelte-preprocess ^4.10.1 development
- tslib ^2.6.0 development
- typescript ^4.5.4 development
- wait-on ^6.0.0 development
- @iconify/svelte ^2.1.1
- chart.js ^3.7.0
- svelte-markdown ^0.2.1
- 484 dependencies
- boto3 *
- requests *