https://github.com/flyteorg/flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
✓
Committers with academic emails
4 of 309 committers (1.3%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (13.4%) to scientific vocabulary

Keywords

data data-analysis data-science dataops declarative fine-tuning flyte golang grpc hacktoberfest kubernetes kubernetes-operator llm machine-learning mlops orchestration-engine production python scale workflow

Keywords from Contributors

distribution parallel llm-inference llm-serving reinforcement-learning large-language-models serving hyperparameter-optimization ray rllib

Last synced: 5 months ago · JSON representation

Repository

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.

Basic Info

Host: GitHub
Owner: flyteorg
License: apache-2.0
Language: Go
Default Branch: master
Homepage: https://flyte.org
Size: 324 MB

Statistics

Stars: 6,461
Watchers: 256
Forks: 745
Open Issues: 200
Releases: 0

Topics

Created over 6 years ago · Last pushed 6 months ago

Metadata Files

Readme Changelog Contributing License Code of conduct Codeowners Security

README.md

Flyte and LF AI & Data Logo

Flyte

:building_construction: :rocket: :chart_with_upwards_trend:

Flyte is an open-source orchestrator that facilitates building production-grade data and ML pipelines. It is built for scalability and reproducibility, leveraging Kubernetes as its underlying platform. With Flyte, user teams can construct pipelines using the Python SDK, and seamlessly deploy them on both cloud and on-premises environments, enabling distributed processing and efficient resource utilization.

Build

Write code in Python or any other language and leverage a robust type engine.

Getting started with Flyte

Deploy & Scale

Either locally or on a remote cluster, execute your models with ease.

* How to contribute

Quick start

Install Flyte's Python SDK bash pip install flytekit
Create a workflow (see example)
Run it locally with: bash pyflyte run hello_world.py hello_world_wf Ready to try a Flyte cluster?
Create a new sandbox cluster, running as a Docker container: bash flytectl demo start
Now execute your workflows on the cluster: bash pyflyte run --remote hello_world.py hello_world_wf

Do you want to see more but don't want to install anything?

Try out the Union platform built on top of Flyte, and get free access to GPUs, data lineage, and more!

Ready to productionize?

Go to the Deployment guide for instructions to install Flyte on different environments

Tutorials

Features

🚀 Strongly typed interfaces: Validate your data at every step of the workflow by defining data guardrails using Flyte types.
🌐 Any language: Write code in any language using raw containers, or choose Python, Java, Scala or JavaScript SDKs to develop your Flyte workflows.
🔒 Immutability: Immutable executions help ensure reproducibility by preventing any changes to the state of an execution.
🧬 Data lineage: Track the movement and transformation of data throughout the lifecycle of your data and ML workflows.
📊 Map tasks: Achieve parallel code execution with minimal configuration using map tasks.
🌎 Multi-tenancy: Multiple users can share the same platform while maintaining their own distinct data and configurations.
🌟 Dynamic workflows: Build flexible and adaptable workflows that can change and evolve as needed, making it easier to respond to changing requirements.
⏯️ Wait for external inputs before proceeding with the execution.
🌳 Branching: Selectively execute branches of your workflow based on static or dynamic data produced by other tasks or input data.
📈 Data visualization: Visualize data, monitor models and view training history through plots.
📂 FlyteFile & FlyteDirectory: Transfer files and directories between local and cloud storage.
🗃️ Structured dataset: Convert dataframes between types and enforce column-level type checking using the abstract 2D representation provided by Structured Dataset.
🛡️ Recover from failures: Recover only the failed tasks.
🔁 Rerun a single task: Rerun workflows at the most granular level without modifying the previous state of a data/ML workflow.
🔍 Cache outputs: Cache task outputs by passing cache=True to the task decorator.
🚩 Intra-task checkpointing: Checkpoint progress within a task execution.
⏰ Timeout: Define a timeout period, after which the task is marked as failure.
🏭 Dev to prod: As simple as changing your domain from development or staging to production.
💸 Spot or preemptible instances: Schedule your workflows on spot instances by setting interruptible to True in the task decorator.
☁️ Cloud-native deployment: Deploy Flyte on AWS, GCP, Azure and other cloud services.
📅 Scheduling: Schedule your data and ML workflows to run at a specific time.
📢 Notifications: Stay informed about changes to your workflow's state by configuring notifications through Slack, PagerDuty or email.
⌛️ Timeline view: Evaluate the duration of each of your Flyte tasks and identify potential bottlenecks.
💨 GPU acceleration: Enable and control your tasks’ GPU demands by requesting resources in the task decorator.
🐳 Dependency isolation via containers: Maintain separate sets of dependencies for your tasks so no dependency conflicts arise.
🔀 Parallelism: Flyte tasks are inherently parallel to optimize resource consumption and improve performance.
💾 Allocate resources dynamically at the task level.

Who's using Flyte

Join the likes of LinkedIn, Spotify, Freenome, Pachama, Warner Bros. and many others in adopting Flyte for mission-critical use cases. For a full list of adopters and information on how to add your organization or project, please visit our ADOPTERS page.

How to stay involved

👥 Monthly community sync: Happening the first Tuesday of every month, this is where the Flyte team provides updates on the project, and community members can share their progress and ask questions.
💬 Slack: Join the Flyte community on Slack to chat with other users, ask questions, and get help.
⚠️ Newsletter: join this group to receive the Flyte Monthly newsletter.
📹 Youtube: Tune into panel discussions, customer success stories, community updates and feature deep dives.
📄 Blog: Here, you can find tutorials and feature deep dives to help you learn more about Flyte.
💡 RFCs: RFCs are used for proposing new ideas and features to improve Flyte. You can refer to them to stay updated on the latest developments and contribute to the growth of the platform.

How to contribute

There are many ways to get involved in Flyte, including:

Submitting bugs and feature requests for various components.
Reviewing the documentation and submitting pull requests for anything from fixing typos to adding new content.
Speaking or writing about Flyte or any other ecosystem integration and letting us know!
Taking on a help wanted or good-first-issue and following the CONTRIBUTING guide to submit changes to the codebase.
Upvoting popular feature requests to show your support.

We :heart: our contributors

License

Flyte is available under the Apache License 2.0. Use it wisely.

Owner

Name: Flyte
Login: flyteorg
Kind: organization
Email: users@flyte.org

Website: https://flyte.org
Twitter: flyteorg
Repositories: 35
Profile: https://github.com/flyteorg

Organization that hosts the Flyte Project with all of the core components. Flyte is an LF AI & Data Graduated Project

Committers

Last synced: 9 months ago

All Time

Total Commits: 4,573
Total Committers: 309
Avg Commits per committer: 14.799
Development Distribution Score (DDS): 0.92

Past Year

Commits: 503
Committers: 101
Avg Commits per committer: 4.98
Development Distribution Score (DDS): 0.879

Top Committers

Name	Email	Commits
Haytham Abuelfutuh	h**m@a**m	368
Dan Rammer	d**l@u**i	243
Katrina Rogan	k**H@g**m	240
Flyte Bot	a**n@f**g	238
goreleaserbot	g**r@c**m	238
Eduardo Apolinario	6****o	208
Kevin Su	p**w@a**g	206
Yuvraj	c**e@e**v	192
pmahindrakar-oss	p**r@g**m	188
Yee Hing Tong	w****r	179
Ketan Umare	1****3	159
Katrina Rogan	k**a@n**o	131
Chang-Hong Hsu	c**u@l**m	111
Jeev B	j****b	108
Ketan Umare	k**e@l**m	98
Future-Outlier	e**1@g**m	88
Niels Bantilan	n**n@g**m	80
Samhita Alla	a**a@g**m	77
Honnix	h****x	62
Andrew Chan	a**n@l**m	59
SandraGH5	8****5	50
Yuvraj	1****t	47
Andrew Dye	a**e@g**m	43
Nikki Everett	n****t	43
Anand Swaminathan	a**n@l**m	43
SmritiSatyanV	9****V	42
David Espejo	8****s	37
Fabio M. Graetz, Ph.D	f**z@g**m	37
Paul Dittamo	3****t	34
Jason Parraga	S**d@g**m	32
and 279 more...

Committer Domains (Top 20 + Academic)

lyft.com: 13 union.ai: 4 apache.org: 3 spotify.com: 2 warlock.ai: 1 synology.com: 1 live.it: 1 onerussian.com: 1 huxuan.org: 1 tomforb.es: 1 projectsomewhere.com: 1 yahoo.co.in: 1 dominodatalab.com: 1 planet.com: 1 morpheusxaut.net: 1 snyk.io: 1 blackshark.ai: 1 brunk.io: 1 chishengliu.com: 1 nuclyde.io: 1 student.uva.nl: 1 umich.edu: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 1,551
Total pull requests: 2,445
Average time to close issues: over 1 year
Average time to close pull requests: about 1 month
Total issue authors: 416
Total pull request authors: 193
Average comments per issue: 3.41
Average comments per pull request: 2.15
Merged pull requests: 1,569
Bot issues: 0
Bot pull requests: 120

Past Year

Issues: 315
Pull requests: 911
Average time to close issues: about 1 month
Average time to close pull requests: 11 days
Issue authors: 128
Pull request authors: 108
Average comments per issue: 1.19
Average comments per pull request: 2.59
Merged pull requests: 581
Bot issues: 0
Bot pull requests: 52

View more stats

Top Authors

Issue Authors

cosmicBboy (81)
wild-endeavor (71)
kumare3 (66)
katrogan (64)
eapolinario (55)
samhita-alla (52)
pingsutw (49)
EngHabu (38)
Sovietaced (35)
pmahindrakar-oss (33)
davidmirror-ops (30)
Future-Outlier (28)
fg91 (23)
hamersaw (23)
RRap0so (18)

Pull Request Authors

eapolinario (292)
Future-Outlier (169)
pingsutw (167)
dependabot[bot] (119)
Sovietaced (117)
wild-endeavor (101)
neverett (95)
flyte-bot (94)
pvditt (74)
hamersaw (71)
katrogan (66)
davidmirror-ops (65)
ddl-ebrown (59)
fg91 (53)
cosmicBboy (41)

Top Labels

Issue Labels

stale (636) bug (593) enhancement (553) untriaged (421) flytekit (250) documentation (187) backlogged (176) housekeeping (137) good first issue (120) ui (100) hacktoberfest (74) flyteadmin (50) plugins (40) flytectl (39) exo (35) help wanted (28) UX (23) waiting for reporter (22) flytekit-java (22) flytepropeller (20) pri3 (20) propeller (16) pri2 (13) helm (11) pri1 (11) fit-and-finish (10) needs-decision (10) ui-redesign (10) flyteremote (8) spec-first (7)

Pull Request Labels

lgtm (527) enhancement (287) documentation (269) size:L (176) size:XS (170) size:S (141) size:M (133) housekeeping (131) dependencies (124) bug (107) go (103) helm (72) size:XXL (64) stale (58) security (56) added (51) size:XL (49) fixed (45) kustomize (22) auto-update-contributors (21) rfc (19) changed (18) hacktoberfest-accepted (14) python (12) review-needed (5) spam (4) boilerplate (3) triage/discuss (3) github_actions (2) rust (2)

Packages

Total packages: 15
Total downloads:
- pypi 308,025 last-month
- npm 1,404 last-month
Total docker downloads: 29,148

Total dependent packages: 97
(may contain duplicates)
Total dependent repositories: 46
(may contain duplicates)
Total versions: 1,373
Total maintainers: 8

pypi.org: flyteidl

IDL for Flyte Platform

Homepage: https://github.com/flyteorg/flyte/tree/master/flyteidl
Documentation: https://flyteidl.readthedocs.io/
License: Apache-2.0
Latest release: 1.16.0
published 7 months ago

Versions: 305
Dependent Packages: 37
Dependent Repositories: 22
Downloads: 286,962 Last month
Docker Downloads: 14,414

Rankings

Dependent packages count: 0.4%

Downloads: 0.5%

Stargazers count: 1.1%

Docker downloads count: 1.5%

Average: 1.5%

Forks count: 2.6%

Dependent repos count: 3.1%

Maintainers (4)

flyte-bot katrogan ytongflyte kumare3

Last synced: 6 months ago

proxy.golang.org: github.com/flyteorg/flyte/flytestdlib

Homepage: https://github.com/flyteorg/flyte
Documentation: https://pkg.go.dev/github.com/flyteorg/flyte/flytestdlib#section-documentation
License: Apache-2.0
Latest release: v1.15.3
published 10 months ago

Versions: 75
Dependent Packages: 17
Dependent Repositories: 5
Docker Downloads: 3,555

Rankings

Stargazers count: 1.1%

Dependent packages count: 1.4%

Forks count: 1.5%

Average: 1.6%

Dependent repos count: 2.3%

Last synced: 5 months ago

proxy.golang.org: github.com/flyteorg/flyte/flyteplugins

Homepage: https://github.com/flyteorg/flyte
Documentation: https://pkg.go.dev/github.com/flyteorg/flyte/flyteplugins#section-documentation
License: Apache-2.0
Latest release: v1.15.3
published 10 months ago

Versions: 75
Dependent Packages: 11
Dependent Repositories: 5
Docker Downloads: 3,555

Rankings

Stargazers count: 1.1%

Forks count: 1.5%

Average: 1.7%

Dependent packages count: 1.9%

Dependent repos count: 2.3%

Last synced: 6 months ago

proxy.golang.org: github.com/flyteorg/flyte/flytepropeller

Homepage: https://github.com/flyteorg/flyte
Documentation: https://pkg.go.dev/github.com/flyteorg/flyte/flytepropeller#section-documentation
License: Apache-2.0
Latest release: v1.15.3
published 10 months ago

Versions: 75
Dependent Packages: 8
Dependent Repositories: 5
Docker Downloads: 3,555

Rankings

Stargazers count: 1.1%

Forks count: 1.5%

Average: 1.8%

Dependent packages count: 2.2%

Dependent repos count: 2.3%

Last synced: 5 months ago

proxy.golang.org: github.com/flyteorg/flyte/flyteadmin

Homepage: https://github.com/flyteorg/flyte
Documentation: https://pkg.go.dev/github.com/flyteorg/flyte/flyteadmin#section-documentation
License: Apache-2.0
Latest release: v1.15.3
published 10 months ago

Versions: 75
Dependent Packages: 3
Dependent Repositories: 3

Rankings

Stargazers count: 1.1%

Forks count: 1.5%

Average: 2.2%

Dependent repos count: 2.9%

Dependent packages count: 3.4%

Last synced: 6 months ago

proxy.golang.org: github.com/flyteorg/flyte/datacatalog

Homepage: https://github.com/flyteorg/flyte
Documentation: https://pkg.go.dev/github.com/flyteorg/flyte/datacatalog#section-documentation
License: Apache-2.0
Latest release: v1.15.3
published 10 months ago

Versions: 75
Dependent Packages: 3
Dependent Repositories: 3

Rankings

Stargazers count: 1.1%

Forks count: 1.5%

Average: 2.2%

Dependent repos count: 2.9%

Dependent packages count: 3.4%

Last synced: 6 months ago

proxy.golang.org: github.com/flyteorg/flyte/flyteidl

Homepage: https://github.com/flyteorg/flyte
Documentation: https://pkg.go.dev/github.com/flyteorg/flyte/flyteidl#section-documentation
License: Apache-2.0
Latest release: v1.16.0
published 7 months ago

Versions: 73
Dependent Packages: 13
Dependent Repositories: 1
Docker Downloads: 3,555

Rankings

Stargazers count: 1.1%

Forks count: 1.5%

Dependent packages count: 1.8%

Average: 2.3%

Dependent repos count: 4.7%

Last synced: 6 months ago

proxy.golang.org: github.com/flyteorg/flyte

Homepage: https://github.com/flyteorg/flyte
Documentation: https://pkg.go.dev/github.com/flyteorg/flyte#section-documentation
License: Apache-2.0
Latest release: v1.15.3
published 10 months ago

Versions: 174
Dependent Packages: 1
Dependent Repositories: 0
Docker Downloads: 138

Rankings

Docker downloads count: 1.0%

Stargazers count: 1.2%

Forks count: 1.5%

Average: 4.3%

Dependent packages count: 8.2%

Dependent repos count: 9.3%

Last synced: 6 months ago

npmjs.org: @flyteorg/flyteidl

Compiled protocol buffers and gRPC service and connect clients/servers for Flyte IDLs

Homepage: https://github.com/flyteorg/flyte#readme
License: Apache-2.0
Latest release: 1.16.0
published 7 months ago

Versions: 248
Dependent Packages: 4
Dependent Repositories: 2
Downloads: 754 Last month
Docker Downloads: 367

Rankings

Docker downloads count: 1.0%

Forks count: 3.7%

Downloads: 4.1%

Dependent packages count: 4.5%

Average: 4.7%

Stargazers count: 7.1%

Dependent repos count: 7.6%

Maintainers (4)

flyte-bot flyteconsole enghabu frank-union.ai

Last synced: 6 months ago

proxy.golang.org: github.com/flyteorg/flyte/docker/sandbox-bundled/bootstrap

Homepage: https://github.com/flyteorg/flyte
Documentation: https://pkg.go.dev/github.com/flyteorg/flyte/docker/sandbox-bundled/bootstrap#section-documentation
License: Apache-2.0
Latest release: v0.0.0-20240124220322-bcb7abef1e3f
published about 2 years ago

Versions: 47
Dependent Packages: 0
Dependent Repositories: 0
Docker Downloads: 9

Rankings

Stargazers count: 1.1%

Forks count: 1.5%

Average: 5.3%

Dependent packages count: 8.5%

Dependent repos count: 10.1%

Last synced: 6 months ago

proxy.golang.org: github.com/flyteorg/flyte/flytectl

Homepage: https://github.com/flyteorg/flyte
Documentation: https://pkg.go.dev/github.com/flyteorg/flyte/flytectl#section-documentation
License: Apache-2.0
Latest release: v0.9.5
published 11 months ago

Versions: 11
Dependent Packages: 0
Dependent Repositories: 0

Rankings

Dependent packages count: 7.0%

Average: 7.4%

Dependent repos count: 7.9%

Last synced: 6 months ago

proxy.golang.org: github.com/flyteorg/flyte/flytecopilot

Homepage: https://github.com/flyteorg/flyte
Documentation: https://pkg.go.dev/github.com/flyteorg/flyte/flytecopilot#section-documentation
License: Apache-2.0
Latest release: v1.15.3
published 10 months ago

Versions: 74
Dependent Packages: 0
Dependent Repositories: 0

Rankings

Dependent packages count: 9.5%

Average: 10.1%

Dependent repos count: 10.7%

Last synced: 6 months ago

npmjs.org: @flyteorg/flyteidl2

Compiled protocol buffers and gRPC service and connect clients/servers for Flyte IDLs

Homepage: https://github.com/flyteorg/flyte#readme
License: Apache-2.0
Latest release: 2.0.0-alpha4
published 6 months ago

Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 64 Last month

Rankings

Dependent repos count: 24.0%

Average: 29.3%

Dependent packages count: 34.6%

Maintainers (4)

flyte-bot flyteconsole enghabu frank-union.ai

Last synced: 6 months ago

npmjs.org: @flyteorg/flyteidl-new

Compiled protocol buffers and gRPC service and connect clients/servers for Flyte IDLs

Homepage: https://github.com/flyteorg/flyte#readme
License: Apache-2.0
Latest release: 0.2.0-b7
published 6 months ago

Versions: 8
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 586 Last month

Rankings

Dependent repos count: 24.0%

Average: 29.3%

Dependent packages count: 34.6%

Maintainers (4)

flyte-bot flyteconsole enghabu frank-union.ai

Last synced: 6 months ago

pypi.org: flyte

Add your description here

Documentation: https://flyte.readthedocs.io/
License: apache-2.0
Latest release: 0.1.0
published 11 months ago

Versions: 57
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 21,063 Last month

Rankings

Dependent packages count: 9.5%

Average: 31.4%

Dependent repos count: 53.4%

Maintainers (2)

flyte-bot unionai-oss

Last synced: 6 months ago

https://github.com/flyteorg/flyte

Science Score: 36.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

Flyte

Build

Deploy & Scale

Get Started · Documentation · Resources

Table of contents

* How to contribute

Quick start

Tutorials

Features

Who's using Flyte

How to stay involved

How to contribute

We :heart: our contributors

License

Owner

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: flyteidl

Rankings

Maintainers (4)

proxy.golang.org: github.com/flyteorg/flyte/flytestdlib

Rankings

proxy.golang.org: github.com/flyteorg/flyte/flyteplugins

Rankings

proxy.golang.org: github.com/flyteorg/flyte/flytepropeller

Rankings

proxy.golang.org: github.com/flyteorg/flyte/flyteadmin

Rankings

proxy.golang.org: github.com/flyteorg/flyte/datacatalog

Rankings

proxy.golang.org: github.com/flyteorg/flyte/flyteidl

Rankings

proxy.golang.org: github.com/flyteorg/flyte

Rankings

npmjs.org: @flyteorg/flyteidl

Rankings

Maintainers (4)

proxy.golang.org: github.com/flyteorg/flyte/docker/sandbox-bundled/bootstrap

Rankings

proxy.golang.org: github.com/flyteorg/flyte/flytectl

Rankings

proxy.golang.org: github.com/flyteorg/flyte/flytecopilot

Rankings

npmjs.org: @flyteorg/flyteidl2

Rankings

Maintainers (4)

npmjs.org: @flyteorg/flyteidl-new

Rankings

Maintainers (4)

pypi.org: flyte

Rankings

Maintainers (2)