Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/cubed-dev/cubed

Scalable array processing with bounded memory

https://github.com/cubed-dev/cubed

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    1 of 13 committers (7.7%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (5.0%) to scientific vocabulary

Keywords

distributed-computing ndimensional-arrays numpy python serverless xarray zarr

Keywords from Contributors

energy-system-model parallel
Last synced: 6 months ago · JSON representation

Repository

Scalable array processing with bounded memory

Basic Info
Statistics
  • Stars: 233
  • Watchers: 8
  • Forks: 21
  • Open Issues: 106
  • Releases: 29
Topics
distributed-computing ndimensional-arrays numpy python serverless xarray zarr
Created almost 4 years ago · Last pushed 6 months ago
Metadata Files
Readme Contributing License

README.md

Cubed

Scalable array processing with bounded memory

Cubed is a Python library for scalable out-of-core multi-dimensional array processing with bounded memory.

  • Cubed provides NumPy and Xarray APIs for processing your multi-dimensional array data
  • Cubed is a drop-in replacement for Dask's Array API
  • Cubed will tell you if your computation would run out of memory before running it
  • Cubed is designed to be robust to failures and will reliably complete a computation
  • Cubed can process hundreds of GB of array data on your laptop using all available cores
  • Cubed is horizontally scalable and stateless, and can scale to multi-TB datasets in the cloud

Documentation

Owner

  • Name: cubed-dev
  • Login: cubed-dev
  • Kind: organization

GitHub Events

Total
  • Create event: 112
  • Release event: 4
  • Issues event: 71
  • Watch event: 85
  • Delete event: 90
  • Issue comment event: 143
  • Push event: 311
  • Pull request event: 208
  • Pull request review event: 37
  • Pull request review comment event: 37
  • Fork event: 5
Last Year
  • Create event: 112
  • Release event: 4
  • Issues event: 71
  • Watch event: 85
  • Delete event: 90
  • Issue comment event: 143
  • Push event: 311
  • Pull request event: 208
  • Pull request review event: 37
  • Pull request review comment event: 37
  • Fork event: 5

Committers

Last synced: 9 months ago

All Time
  • Total Commits: 771
  • Total Committers: 13
  • Avg Commits per committer: 59.308
  • Development Distribution Score (DDS): 0.049
Past Year
  • Commits: 187
  • Committers: 9
  • Avg Commits per committer: 20.778
  • Development Distribution Score (DDS): 0.07
Top Committers
Name Email Commits
Tom White t****e@g****m 733
Tom Nicholas t****s@c****u 14
pre-commit-ci[bot] 6****] 6
Alex Merose al@m****m 5
rsignell 1****l 3
Raphael Hagen n****n@g****m 2
Ryan Avery r****y@R****l 2
Wong Songhan s****9@g****m 1
Timothy Hodson 3****s 1
Ryan Avery r****y@g****m 1
Pau Balanzà 9****4 1
Josep Sampé j****e@g****m 1
Davin Potts a****o 1
Committer Domains (Top 20 + Academic)
merose.com: 1 columbia.edu: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 201
  • Total pull requests: 472
  • Average time to close issues: 5 months
  • Average time to close pull requests: 9 days
  • Total issue authors: 17
  • Total pull request authors: 14
  • Average comments per issue: 1.38
  • Average comments per pull request: 0.42
  • Merged pull requests: 392
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 55
  • Pull requests: 241
  • Average time to close issues: 12 days
  • Average time to close pull requests: 3 days
  • Issue authors: 9
  • Pull request authors: 4
  • Average comments per issue: 0.58
  • Average comments per pull request: 0.26
  • Merged pull requests: 198
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • tomwhite (142)
  • TomNicholas (29)
  • alxmrs (7)
  • norlandrhagen (4)
  • rbavery (2)
  • dcherian (2)
  • rabernat (2)
  • songhan89 (2)
  • wietzesuijker (2)
  • balanz24 (2)
  • Yefee (1)
  • rgommers (1)
  • applio (1)
  • TomAugspurger (1)
  • abourramouss (1)
Pull Request Authors
  • tomwhite (431)
  • alxmrs (13)
  • songhan89 (4)
  • applio (4)
  • wietzesuijker (4)
  • thodson-usgs (3)
  • norlandrhagen (2)
  • dcherian (2)
  • rbavery (2)
  • TomNicholas (2)
  • JosepSampe (2)
  • rsignell (1)
  • TomAugspurger (1)
  • balanz24 (1)
Top Labels
Issue Labels
runtime (37) array api (30) core (22) documentation (18) zarr (18) optimization (14) bug (11) storage (10) enhancement (9) xarray-integration (9) primitive (8) memory (8) help wanted (6) single-machine (6) tools (5) good first issue (4) examples (4) icechunk 🧊 (3) upstream (3) benchmarks (3) planning (2) gpu (2)
Pull Request Labels
documentation (49) array api (45) runtime (37) bug (25) zarr (24) examples (16) optimization (16) core (16) memory (11) primitive (10) xarray-integration (9) upstream (9) storage (8) tests (8) planning (5) tools (4) icechunk 🧊 (4) single-machine (3) diagnostics (1)

Dependencies

requirements.txt pypi
  • apache-beam *
  • dask *
  • fsspec *
  • gcsfs *
  • lithops *
  • numpy >=1.22
  • pydot *
  • pytest *
  • pytest-mock *
  • rechunker *
  • s3fs *
  • tenacity *
  • zarr <2.12
.github/workflows/array-api-tests.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
.github/workflows/docs.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
  • ad-m/github-push-action master composite
  • sphinx-notes/pages v2 composite
.github/workflows/modal-tests.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
.github/workflows/scale-tests.yml.disabled actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
  • actions/upload-artifact v3 composite
  • microsoft/variable-substitution v1 composite
.github/workflows/tests.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
docs/requirements.txt pypi
  • apache-beam *
  • dask *
  • fsspec *
  • lithops >=2.7.0
  • myst-parser *
  • numpy >=1.22
  • pydot *
  • pytest *
  • pytest-cov *
  • pytest-mock *
  • rechunker *
  • s3fs *
  • sphinx-autobuild *
  • sphinx-book-theme *
  • tenacity *
  • tqdm *
  • zarr *
.github/workflows/dask-tests.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
  • ts-graphviz/setup-graphviz v1 composite
.github/workflows/mypy.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
.github/workflows/release.yml actions
  • actions/checkout v3 composite
  • actions/download-artifact v3 composite
  • actions/setup-python v3 composite
  • actions/upload-artifact v3 composite
  • pypa/gh-action-pypi-publish release/v1 composite
examples/coiled/aws/requirements.txt pypi
  • coiled *
  • cubed *
  • s3fs *
  • tqdm *
examples/dataflow/requirements.txt pypi
  • apache_beam *
  • cubed *
  • gcsfs *
examples/dataflow/setup.py pypi
  • cubed *
examples/lithops/aws-lambda/requirements.txt pypi
  • cubed *
  • lithops *
  • s3fs *
  • tqdm *
examples/lithops/gcf/requirements.txt pypi
  • cubed *
  • gcsfs *
  • lithops *
  • tqdm *
examples/modal/aws/requirements.txt pypi
  • cubed *
  • s3fs *
  • tqdm *
examples/modal/gcp/requirements.txt pypi
  • cubed *
  • gcsfs *
  • tqdm *
pyproject.toml pypi
  • aiostream *
  • fsspec *
  • mypy_extensions *
  • networkx < 2.8.3
  • numpy >= 1.22
  • tenacity *
  • toolz *
  • zarr *
setup.py pypi
.github/workflows/slow-tests.yml.disabled actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite