Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/arogozhnikov/r2_fsspec

Support of r2 storage in fsspec as r2://

https://github.com/arogozhnikov/r2_fsspec

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (5.9%) to scientific vocabulary
Last synced: 9 months ago · JSON representation

Repository

Support of r2 storage in fsspec as r2://

Basic Info
  • Host: GitHub
  • Owner: arogozhnikov
  • License: mit
  • Language: Python
  • Default Branch: main
  • Size: 11.7 KB
Statistics
  • Stars: 1
  • Watchers: 1
  • Forks: 0
  • Open Issues: 0
  • Releases: 1
Created over 1 year ago · Last pushed about 1 year ago
Metadata Files
Readme License

README.md

r2_fsspec

Support of r2 storage in fsspec as r2://

Cloudflare's R2 has S3-like interface.

There are a couple of reasons for existense of this package:

  1. use prefix r2://bucket/path/to-object not s3://

    • very helpful when using s3 and r2 in parallel

  2. this package auto-grabs credentials for r2 from file, pointed by R2_FSSPEC_CREDENTIALS env var if specified, otherwise falls back to s3fs to search for credentials.

  3. ~~strategy of s3fs upload does not work with R2, so this package is pinned on specific version of modified fsspec.~~ I've upstreamed necessary changes, s3fs now supports R2 directly

  4. by default, r2_fsspec prevents file overwrites (unlike s3fs and unlike fsspec in general).

This package heavily relies on canonical fsspec/s3fs, and patches necessary places, thus it dependes on strictly pinned version of s3fs.

Usage:

pip install git+https://github.com/arogozhnikov/r2_fsspec

you don't need to import r2_fsspec directly, fsspec will automatically use it to handle links with 'r2://'

Owner

  • Name: Alex Rogozhnikov
  • Login: arogozhnikov
  • Kind: user
  • Location: San Francisco
  • Company: Aperture Science

ML + Science, einops, scientific tools

GitHub Events

Total
  • Watch event: 1
  • Public event: 1
  • Push event: 6
  • Create event: 1
Last Year
  • Watch event: 1
  • Public event: 1
  • Push event: 6
  • Create event: 1

Issues and Pull Requests

Last synced: about 1 year ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

pyproject.toml pypi
  • s3fs ==2024.10.0