dfir_datalog

A high-level distributed programming framework for Rust

https://github.com/hydro-project/hydro

Science Score: 54.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • CITATION.cff file
    Found CITATION.cff file
  • codemeta.json file
    Found codemeta.json file
  • .zenodo.json file
    Found .zenodo.json file
  • DOI references
  • Academic publication links
    Links to: arxiv.org
  • Academic email domains
  • Institutional organization owner
  • JOSS paper metadata
  • Scientific vocabulary similarity
    Low similarity (10.5%) to scientific vocabulary

Keywords

distributed-systems rust
Last synced: 6 months ago · JSON representation ·

Repository

A high-level distributed programming framework for Rust

Basic Info
  • Host: GitHub
  • Owner: hydro-project
  • License: apache-2.0
  • Language: Rust
  • Default Branch: main
  • Homepage: https://hydro.run/
  • Size: 316 MB
Statistics
  • Stars: 986
  • Watchers: 14
  • Forks: 58
  • Open Issues: 129
  • Releases: 0
Topics
distributed-systems rust
Created over 4 years ago · Last pushed 6 months ago
Metadata Files
Readme Contributing License Citation

README.md

"hf"

Crates.io Docs.rs

Hydro is a high-level distributed programming framework for Rust. Hydro can help you quickly write scalable distributed services that are correct by construction. Much like Rust helps with memory safety, Hydro helps with distributed safety.

Hydro integrates naturally with standard Rust constructs and IDEs, providing types and programming constructs for ensuring distributed safety. Under the covers, Hydro is powered by the Dataflow Intermediate Representation (DFIR), a compiler and low-level runtime for stream processing. DFIR enables automatic vectorization and efficient scheduling without restricting your application logic.

Get started today at hydro.run!

Learn More

Contributing

For Hydro development setup and contribution info, see CONTRIBUTING.md.

Owner

  • Name: Hydro
  • Login: hydro-project
  • Kind: organization
  • Location: Berkeley, CA

Citation (CITATION.cff)

# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!

cff-version: 1.2.0
title: Hydro
message: >-
  If you use this software, please cite it using the
  metadata from this file.
type: software
authors:
  - given-names: Mingwei
    family-names: Samuel
    email: mingwei@shv.com
    affiliation: Sutter Hill Ventures
    orcid: 'https://orcid.org/0009-0004-9873-6266'
  - given-names: Shadaj
    family-names: Laddad
    email: shadaj@berkeley.edu
    orcid: 'https://orcid.org/0000-0002-6658-6548'
    affiliation: UC Berkeley
  - given-names: Joseph M.
    family-names: Hellerstein
    email: hellerstein@berkeley.edu
    orcid: 'https://orcid.org/0000-0002-7712-4306'
    affiliation: UC Berkeley
  - given-names: Rohit
    family-names: Kulshreshtha
    email: rohit@shv.com
    affiliation: Sutter Hill Ventures
  - given-names: Lucky
    family-names: Katahanas
    email: lucky@shv.com
    affiliation: Sutter Hill Ventures
  - given-names: Justin
    family-names: Jaffray
    email: justin@shv.com
    affiliation: Sutter Hill Ventures
  - given-names: David
    family-names: Chu
    affiliation: UC Berkeley
    email: thedavidchu@berkeley.edu
  - given-names: Conor
    email: conorpower@berkeley.edu
    family-names: Power
    orcid: 'https://orcid.org/0000-0002-0660-5110'
    affiliation: UC Berkeley
repository-code: 'https://github.com/hydro-project/hydro/'
url: 'https://hydro.run/'
abstract: >-
  Hydro is a framework for programming distributed systems,
  built on top of DFIR, a lattice-aware dataflow system.
keywords:
  - dataflow
  - distributed systems
  - cloud computing
  - query optimization
  - programming languages
  - rust
license: Apache-2.0

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 103
  • Total pull requests: 255
  • Average time to close issues: about 2 months
  • Average time to close pull requests: 7 days
  • Total issue authors: 8
  • Total pull request authors: 15
  • Average comments per issue: 0.51
  • Average comments per pull request: 0.68
  • Merged pull requests: 164
  • Bot issues: 0
  • Bot pull requests: 2
Past Year
  • Issues: 101
  • Pull requests: 254
  • Average time to close issues: about 1 month
  • Average time to close pull requests: 3 days
  • Issue authors: 8
  • Pull request authors: 15
  • Average comments per issue: 0.5
  • Average comments per pull request: 0.68
  • Merged pull requests: 164
  • Bot issues: 0
  • Bot pull requests: 2
Top Authors
Issue Authors
  • shadaj (43)
  • MingweiSamuel (38)
  • jhellerstein (12)
  • rohitkulshreshtha (4)
  • davidchuyaya (3)
  • rbalicki2 (1)
  • ntrp (1)
  • chenyh1990 (1)
Pull Request Authors
  • shadaj (115)
  • MingweiSamuel (74)
  • davidchuyaya (21)
  • jhellerstein (15)
  • Copilot (10)
  • luckyworkama (9)
  • Honeycomb (2)
  • Flamenco (2)
  • rohitkulshreshtha (1)
  • github-actions[bot] (1)
  • rupert648 (1)
  • runllm-pr-agent[bot] (1)
  • FHaggs (1)
  • ydewit (1)
  • RyanAlameddine (1)
Top Labels
Issue Labels
documentation (14) hydro lang (14) deploy (12) dfir_lang syntax (12) enhancement (9) Small (8) debugging/observability (8) testing (6) dfir_rs scheduler (6) bug (6) P2 (4) dfir_rs runtime (core api) (4) performance (4) P0 (3) good first issue (3) P1 (3) meta (3) staging (2) question (1) stageleft (1) hydroflow+ (1) wontfix (1) dfir syntax (1) Large (1)
Pull Request Labels
bug (1)

Packages

  • Total packages: 20
  • Total downloads:
    • cargo 258,429 total
  • Total dependent packages: 7
    (may contain duplicates)
  • Total dependent repositories: 4
    (may contain duplicates)
  • Total versions: 131
  • Total maintainers: 2
crates.io: variadics

Variadic generics on stable Rust using tuple lists

  • Versions: 10
  • Dependent Packages: 2
  • Dependent Repositories: 1
  • Downloads: 36,723 Total
Rankings
Dependent packages count: 12.2%
Dependent repos count: 16.7%
Average: 26.3%
Downloads: 49.8%
Maintainers (2)
Last synced: 6 months ago
crates.io: lattices

Lattice data types for simplifying distributed state by providing associativity, commutativity, and idempotence.

  • Versions: 19
  • Dependent Packages: 1
  • Dependent Repositories: 1
  • Downloads: 45,323 Total
Rankings
Dependent repos count: 16.5%
Dependent packages count: 18.2%
Average: 27.7%
Downloads: 48.6%
Maintainers (2)
Last synced: 6 months ago
crates.io: pusherator

Push-based version of Rust iterators

  • Versions: 13
  • Dependent Packages: 1
  • Dependent Repositories: 1
  • Downloads: 38,349 Total
Rankings
Dependent repos count: 16.7%
Dependent packages count: 18.1%
Average: 31.2%
Downloads: 58.8%
Maintainers (2)
Last synced: 6 months ago
crates.io: hydro_optimize

Automatic decoupling and partitioning

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 248 Total
Rankings
Stargazers count: 5.7%
Forks count: 8.7%
Dependent repos count: 20.6%
Dependent packages count: 27.3%
Average: 31.4%
Downloads: 94.6%
Maintainers (1)
Last synced: 6 months ago
crates.io: hydro_deploy_integration

`hydro_deploy` integration for deploying Rust crates.

  • Versions: 3
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 1,573 Total
Rankings
Stargazers count: 6.6%
Forks count: 9.8%
Dependent repos count: 22.7%
Dependent packages count: 30.1%
Average: 33.1%
Downloads: 96.1%
Maintainers (1)
Last synced: 6 months ago
crates.io: dfir_lang

Hydro's Dataflow Intermediate Representation (DFIR) implementation

  • Versions: 5
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 3,892 Total
Rankings
Stargazers count: 7.5%
Forks count: 9.8%
Dependent repos count: 23.9%
Dependent packages count: 31.7%
Average: 33.7%
Downloads: 95.4%
Maintainers (2)
Last synced: 6 months ago
crates.io: dfir_datalog_core

Datalog implementation for DFIR

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 3,375 Total
Rankings
Stargazers count: 7.5%
Forks count: 9.8%
Dependent repos count: 23.9%
Dependent packages count: 31.7%
Average: 33.7%
Downloads: 95.4%
Maintainers (2)
Last synced: 6 months ago
crates.io: dfir_rs

DFIR runtime for Rust, used by Hydro.

  • Versions: 5
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 24,545 Total
Rankings
Stargazers count: 7.5%
Forks count: 9.8%
Dependent repos count: 23.9%
Dependent packages count: 31.7%
Average: 33.7%
Downloads: 95.4%
Maintainers (2)
Last synced: 6 months ago
crates.io: dfir_macro

Macro for using Hydro's Data Flow Intermediate Representation (DFIR).

  • Versions: 5
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 3,557 Total
Rankings
Stargazers count: 7.5%
Forks count: 9.8%
Dependent repos count: 23.9%
Dependent packages count: 31.7%
Average: 33.7%
Downloads: 95.4%
Maintainers (2)
Last synced: 6 months ago
crates.io: dfir_datalog

Datalog proc-macro for DFIR

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 3,292 Total
Rankings
Stargazers count: 7.5%
Forks count: 9.8%
Dependent repos count: 23.9%
Dependent packages count: 31.7%
Average: 33.7%
Downloads: 95.4%
Maintainers (2)
Last synced: 6 months ago
crates.io: multiplatform_test

A simple attribute macro to combine `#[test]` and `#[wasm_bindgen_test]`

  • Versions: 6
  • Dependent Packages: 1
  • Dependent Repositories: 1
  • Downloads: 9,743 Total
Rankings
Dependent repos count: 16.7%
Dependent packages count: 18.2%
Average: 36.8%
Downloads: 75.4%
Maintainers (2)
Last synced: 6 months ago
crates.io: example_test

Test `cargo example` code.

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 255 Total
Rankings
Dependent repos count: 20.6%
Dependent packages count: 27.3%
Average: 47.5%
Downloads: 94.6%
Maintainers (1)
Last synced: 6 months ago
crates.io: include_mdtests

Include markdown for testing, via doctests.

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 655 Total
Rankings
Dependent repos count: 22.7%
Dependent packages count: 30.1%
Average: 49.6%
Downloads: 96.1%
Maintainers (1)
Last synced: 6 months ago
crates.io: hydro_lang

Choreographic programming for Hydro, built on DFIR

  • Versions: 7
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 3,589 Total
Rankings
Dependent repos count: 23.9%
Dependent packages count: 31.7%
Average: 50.4%
Downloads: 95.4%
Maintainers (2)
Last synced: 6 months ago
crates.io: hydro_std

Standard library of distributed systems building blocks for Hydro

  • Versions: 5
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 2,610 Total
Rankings
Dependent repos count: 23.9%
Dependent packages count: 31.7%
Average: 50.4%
Downloads: 95.4%
Maintainers (2)
Last synced: 6 months ago
crates.io: variadics_macro

Procedural macros for the `variadics` crate.

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 24,381 Total
Rankings
Dependent repos count: 24.6%
Dependent packages count: 32.7%
Average: 51.1%
Downloads: 96.0%
Maintainers (1)
Last synced: 6 months ago
crates.io: hydroflow_deploy_integration

`hydro_deploy` integration for Hydroflow

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 3,529 Total
Rankings
Dependent repos count: 25.7%
Dependent packages count: 34.1%
Average: 52.2%
Downloads: 96.7%
Maintainers (1)
Last synced: 6 months ago
crates.io: lattices_macro

Procedural macros for the `lattices` crate.

  • Versions: 6
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 26,737 Total
Rankings
Dependent repos count: 26.5%
Dependent packages count: 35.1%
Average: 52.4%
Downloads: 95.6%
Maintainers (1)
Last synced: 6 months ago
crates.io: hydro_cli

Hydro Deploy Command Line Interface

  • Versions: 16
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 15,312 Total
Rankings
Dependent repos count: 28.6%
Dependent packages count: 33.3%
Average: 53.0%
Downloads: 97.2%
Maintainers (2)
Last synced: 6 months ago
crates.io: hydro_deploy

Hydro Deploy

  • Versions: 12
  • Dependent Packages: 2
  • Dependent Repositories: 0
  • Downloads: 10,741 Total
Rankings
Dependent repos count: 29.6%
Dependent packages count: 34.9%
Average: 53.7%
Downloads: 96.5%
Maintainers (2)
Last synced: 6 months ago

Dependencies

Cargo.lock cargo
  • 164 dependencies
hydro_cli/Cargo.toml cargo
  • anyhow 1.0.69
  • async-channel 1.8.0
  • async-process 1.6.0
  • async-recursion 1
  • async-ssh2-lite 0.4.2
  • async-trait 0.1.64
  • bytes 1.1.0
  • cargo_metadata 0.15.4
  • clap 4.1.8
  • ctrlc 3.2.5
  • dyn-clone 1
  • futures 0.3.26
  • futures-core 0.3.26
  • hydroflow_cli_integration ^0.3.0
  • indicatif 0.17.3
  • nanoid 0.4.0
  • nix 0.26.2
  • once_cell 1.17
  • pyo3 0.18
  • pyo3-asyncio 0.18
  • pythonize 0.18
  • serde 1
  • serde_json 1
  • shell-escape 0.1.5
  • tempfile 3.3.0
  • tokio 1.16
  • tokio-util 0.7.7
hydro_cli_examples/Cargo.toml cargo
hydroflow/Cargo.toml cargo
  • chrono 0.4.20 development
  • clap 4.1.8 development
  • colored 2.0 development
  • core_affinity 0.5.10 development
  • ctor 0.2 development
  • futures 0.3 development
  • hdrhistogram 7 development
  • insta 1.7.1 development
  • multiplatform_test 0.0.0 development
  • rand 0.8.4 development
  • rand_distr 0.4.3 development
  • regex 1 development
  • static_assertions 1.1.0 development
  • textnonce 1.0.0 development
  • time 0.3 development
  • tracing-subscriber 0.3 development
  • trybuild 1.0.80 development
  • wasm-bindgen-test 0.3.34 development
  • zipf 7.0.0 development
  • bincode 1.3
  • byteorder 1.4.3
  • bytes 1.1.0
  • futures 0.3
  • hydroflow_cli_integration ^0.3.0
  • hydroflow_datalog ^0.5.0
  • hydroflow_lang ^0.5.0
  • hydroflow_macro ^0.5.0
  • instant 0.1.12
  • itertools 0.10
  • lattices ^0.5.0
  • pusherator ^0.0.3
  • pyo3 0.18
  • ref-cast 1.0
  • regex 1.8.4
  • rustc-hash 1.1.0
  • sealed 0.5
  • serde 1
  • serde_json 1
  • slotmap 1.0
  • smallvec 1.10.0
  • tokio-stream 0.1.10
  • tracing 0.1
  • variadics ^0.0.2
hydroflow_cli_integration/Cargo.toml cargo
hydroflow_datalog/Cargo.toml cargo
hydroflow_datalog_core/Cargo.toml cargo
  • insta 1.7.1 development
  • prettyplease 0.2.0 development
  • tempfile 3.2.0 development
  • hydroflow_lang ^0.5.0
  • proc-macro-crate 1.1.0
  • proc-macro2 1.0.63
  • quote 1.0.0
  • rust-sitter 0.3.2
  • slotmap 1.0.6
  • syn 2.0.0
hydroflow_lang/Cargo.toml cargo
  • auto_impl 1.0.1
  • clap 4.1.8
  • data-encoding 2.4.0
  • itertools 0.10
  • prettyplease 0.2.0
  • proc-macro2 1.0.63
  • quote 1.0.0
  • regex 1.7.0
  • serde 1.0.1
  • serde_json 1.0.8
  • slotmap 1.0.6
  • syn 2.0.0
  • webbrowser 0.8.11
hydroflow_macro/Cargo.toml cargo
lattices/Cargo.toml cargo
multiplatform_test/Cargo.toml cargo
  • env_logger 0.10 development
  • log 0.4 development
  • tracing 0.1 development
  • tracing-subscriber 0.3 development
  • wasm-bindgen-test 0.3 development
  • proc-macro2 1.0.63
  • quote 1.0
pusherator/Cargo.toml cargo
relalg/Cargo.toml cargo
variadics/Cargo.toml cargo
website_playground/Cargo.toml cargo
  • wasm-bindgen-test 0.3.13 development
  • console_error_panic_hook 0.1.6
  • futures 0.3
  • prettyplease 0.2.0
  • proc-macro2 1.0.63
  • quote 1.0
  • serde 1.0
  • serde-wasm-bindgen 0.4
  • syn 2.0.0
  • tokio 1.16
  • wasm-bindgen 0.2
  • web-sys 0.3.61
  • wee_alloc 0.4.5
hydro_cli/pyproject.toml pypi
hydro_cli/python_tests/requirements.txt pypi
  • pytest * test
  • pytest-asyncio * test
.github/workflows/audit.yml actions
  • actions-rs/audit-check v1 composite
  • actions/checkout v3 composite
.github/workflows/build-cli.yml actions
  • PyO3/maturin-action v1 composite
  • actions/checkout v3 composite
  • actions/download-artifact v3 composite
  • actions/setup-python v4 composite
  • actions/upload-artifact v3 composite
  • fkirc/skip-duplicate-actions v5.3.0 composite
.github/workflows/ci.yml actions
  • actions-rs/cargo v1 composite
  • actions-rs/toolchain v1 composite
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
  • actions/upload-artifact master composite
  • benchmark-action/github-action-benchmark v1 composite
  • fkirc/skip-duplicate-actions v5.3.0 composite
.github/workflows/conventional_commits.yml actions
  • actions/checkout v2 composite
  • webiny/action-conventional-commits v1.1.0 composite
.github/workflows/examples-container.yml actions
  • actions/checkout v3 composite
  • docker/build-push-action v3 composite
  • docker/login-action v2 composite
  • docker/setup-buildx-action v2 composite
  • docker/setup-qemu-action v2 composite
  • fkirc/skip-duplicate-actions v5.3.0 composite
.github/workflows/release.yml actions
  • actions-rs/cargo v1 composite
  • actions-rs/toolchain v1 composite
  • actions/checkout v3 composite
  • tibdex/github-app-token v1 composite
Cargo.toml cargo
benches/Cargo.toml cargo
  • criterion 0.4 development
  • differential-dataflow * development
  • lazy_static 1.4.0 development
  • rand 0.8.4 development
  • rand_distr 0.4.3 development
  • seq-macro 0.2 development
  • static_assertions 1.1.0 development
  • timely * development
  • tokio 1.0 development
Dockerfile docker
  • rust slim-buster build
docs/package-lock.json npm
  • 1131 dependencies
docs/package.json npm
  • @docusaurus/module-type-aliases 2.4.1 development
  • @docusaurus/core 2.4.1
  • @docusaurus/plugin-ideal-image ^2.4.1
  • @docusaurus/preset-classic 2.4.1
  • @docusaurus/theme-mermaid ^2.4.1
  • @mdx-js/react ^1.6.22
  • @monaco-editor/react ^4.4.6
  • clsx ^1.2.1
  • hast-util-is-element ^1.1.0
  • mermaid ^9.3.0
  • prism-react-renderer ^1.3.5
  • raw-loader ^4.0.2
  • react ^17.0.2
  • react-dom ^17.0.2
  • react-katex ^3.0.1
  • rehype-katex ^5.0.0
  • remark-math ^3.0.1
  • website_playground file:../website_playground/pkg