https://github.com/awslabs/mountpoint-s3
A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (15.3%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.
Basic Info
Statistics
- Stars: 5,367
- Watchers: 50
- Forks: 212
- Open Issues: 118
- Releases: 32
Topics
Metadata Files
README.md
Mountpoint for Amazon S3
Mountpoint for Amazon S3 is a simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system. With Mountpoint for Amazon S3, your applications can access objects stored in Amazon S3 through file operations like open and read. Mountpoint for Amazon S3 automatically translates these operations into S3 object API calls, giving your applications access to the elastic storage and throughput of Amazon S3 through a file interface.
Mountpoint for Amazon S3 is optimized for applications that need high read throughput to large objects, potentially from many clients at once, and to write new objects sequentially from a single client at a time. This means it's a great fit for applications that use a file interface to:
* read large objects from S3, potentially from many instances concurrently, without downloading them to local storage first
* access only some S3 objects out of a larger data set, but can't predict which objects in advance
* upload their output to S3 directly, or upload files from local storage with tools like cp
but probably not the right fit for applications that:
* use file operations that S3 doesn't natively support, like directory renaming or symlinks
* make edits to existing files (don't work on your Git repository or run vim in Mountpoint 😄)
Mountpoint for Amazon S3 does not implement all the features of a POSIX file system, and there are some differences that may affect compatibility with your application. See Mountpoint file system behavior for a detailed description of Mountpoint for Amazon S3's behavior and POSIX support and how they could affect your application. To troubleshoot file operations that may not be supported by Mountpoint, see the troubleshooting documentation.
Current status
Mountpoint for Amazon S3 is generally available! We're tracking future feature development on the Mountpoint for Amazon S3 public roadmap. We're always interested in feedback on features, performance, and compatibility. Please send feedback by opening a new GitHub issue or adding your input to an existing roadmap issue.
Notices
[!IMPORTANT] Mountpoint for Amazon S3 v1.4.0, released on January 26, 2024, contains an issue that causes intermittent read failures. We recommend that customers using v1.4.0 upgrade to v1.4.1 or later.
Getting started
Run these two commands to install Mountpoint for Amazon S3 on your Amazon Linux EC2 instance (for Graviton instances, replace x86_64 with arm64 in the URL):
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
sudo yum install -y ./mount-s3.rpm
On Ubuntu, use these commands instead (for Graviton instances, replace x86_64 with arm64 in the URL):
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.deb
sudo apt-get install -y ./mount-s3.deb
[!NOTE] See Installing Mountpoint for Amazon S3 for detailed instructions and other installation options, including the Mountpoint for Amazon S3 CSI driver for Kubernetes applications, or building from source.
Once you've got Mountpoint for Amazon S3 installed, you can mount your Amazon S3 bucket.
You'll need valid AWS credentials to access your bucket.
Mountpoint will automatically use credentials from an IAM role associated with your EC2 instance,
or the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables,
or from several other sources.
To mount your bucket, run this command,
replacing amzn-s3-demo-bucket with the name of your bucket
and /path/to/mount with the directory you want to mount the bucket to:
mount-s3 amzn-s3-demo-bucket /path/to/mount
Now you can work with your bucket contents as if they were a local file system:
ls /path/to/mount
echo "Hello World!" > /path/to/mount/Data.txt
cat /path/to/mount/Data.txt
When you're finished accessing your bucket, you can unmount it (you might need sudo):
umount /path/to/mount
[!NOTE] See Configuring Mountpoint for Amazon S3 for more details on how to configure and use Mountpoint.
Compatibility with other storage services
Mountpoint for Amazon S3 is designed for high-performance access to the Amazon S3 service. While it may be functional against other storage services that use S3-like APIs, we aren't able to provide support for those use cases, and they may inadvertently break when we make changes to better support Amazon S3. We welcome contributions of minor compatibility fixes or performance improvements for these services if the changes can be tested against Amazon S3.
Contributing
We welcome contributions to Mountpoint for Amazon S3! Please see CONTRIBUTING.md for more information on how to report bugs or submit pull requests. We especially welcome contributions to issues tagged as good first issues to work on.
For potential bug reports, see LOGGING.md for details on how to capture logging data. For performance issues, see BENCHMARKING.md for documentation about performance regression testing.
Security
If you discover a potential security issue in this project we ask that you notify AWS Security via our vulnerability reporting page. Please do not create a public GitHub issue.
Code of conduct
This project has adopted the Amazon Open Source Code of Conduct. See CODEOFCONDUCT.md for more details.
License
This project is licensed under the Apache-2.0 License. It builds on a number of other awesome projects with open source licenses, primarily: * The fuser bindings for FUSE (MIT license) * The AWS Common Runtime for interacting with S3 (Apache 2.0 license)
See deny.toml for a list of licenses used by our dependencies.
Owner
- Name: Amazon Web Services - Labs
- Login: awslabs
- Kind: organization
- Location: Seattle, WA
- Website: http://amazon.com/aws/
- Repositories: 914
- Profile: https://github.com/awslabs
AWS Labs
Committers
Last synced: 9 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| James Bornholt | b****t@a****m | 228 |
| Daniel Carl Jones | d****a@a****m | 197 |
| Alessandro Passaro | a****x@a****k | 167 |
| Monthon Klongklaew | m****k@a****m | 126 |
| Luke Nelson | l****l@a****m | 67 |
| Volodkin Vladislav | v****n@g****m | 65 |
| Ankit Saurabh | s****k@a****k | 34 |
| Simon Beal | 5****h | 26 |
| Burak Varlı | b****r@a****k | 20 |
| dependabot[bot] | 4****] | 12 |
| Rajeev Joshi | 4****v | 12 |
| Christian Hagemeier | c****m@a****m | 11 |
| Andres Santana | h****a@a****m | 9 |
| Eugene Dolgy | 4****o | 7 |
| Charles | 1****a | 6 |
| ahmarsuhail | a****l@g****m | 6 |
| Sahitya Damera | s****d@a****m | 5 |
| Andy Peace | a****e@a****m | 4 |
| Renan Magagnin | r****n@g****m | 4 |
| Rajeev Joshi | {****} | 3 |
| Tadiwa Magwenzi | 8****n | 3 |
| Josh Chorlton | j****l | 3 |
| rajdchak | r****k@a****k | 2 |
| ericjheinz | 6****z | 2 |
| andrewatamzn | 1****n | 2 |
| Isaev Ilya | i****v@a****k | 2 |
| Rostyslav | r****n@g****m | 1 |
| Rich Bowen | r****n@r****m | 1 |
| Paulo Miguel Almeida | p****s@g****m | 1 |
| Jiri Pospisil | j****i@j****m | 1 |
| and 15 more... | ||
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 335
- Total pull requests: 1,454
- Average time to close issues: about 2 months
- Average time to close pull requests: 8 days
- Total issue authors: 211
- Total pull request authors: 51
- Average comments per issue: 2.5
- Average comments per pull request: 0.34
- Merged pull requests: 1,075
- Bot issues: 0
- Bot pull requests: 25
Past Year
- Issues: 93
- Pull requests: 803
- Average time to close issues: 23 days
- Average time to close pull requests: 5 days
- Issue authors: 75
- Pull request authors: 23
- Average comments per issue: 1.39
- Average comments per pull request: 0.19
- Merged pull requests: 572
- Bot issues: 0
- Bot pull requests: 7
Top Authors
Issue Authors
- dannycjones (31)
- jamesbornholt (23)
- mhnap (9)
- tchaton (8)
- jchorl (7)
- monthonk (7)
- sauraank (6)
- mura890 (4)
- thundergolfer (4)
- picoroma (4)
- passaro (4)
- at16cib (3)
- pawalt (3)
- nguyenminhdungpg (3)
- bradyforcier1 (3)
Pull Request Authors
- dannycjones (301)
- passaro (297)
- vladem (140)
- jamesbornholt (120)
- monthonk (106)
- muddyfish (78)
- c-hagem (76)
- sauraank (62)
- unexge (54)
- ujinho (35)
- dependabot[bot] (25)
- sahityadg (21)
- arsh (20)
- adpeace (14)
- renanmagagnin (12)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 8
-
Total downloads:
- cargo 169,913 total
- Total docker downloads: 28
-
Total dependent packages: 3
(may contain duplicates) -
Total dependent repositories: 3
(may contain duplicates) - Total versions: 125
- Total maintainers: 6
proxy.golang.org: github.com/awslabs/mountpoint-s3
- Homepage: https://github.com/awslabs/mountpoint-s3
- Documentation: https://pkg.go.dev/github.com/awslabs/mountpoint-s3#section-documentation
- License: Apache-2.0
-
Latest release: v1.19.0
published 8 months ago
Rankings
crates.io: mountpoint-s3-crt-sys
Rust FFI bindings to the AWS Common Runtime for Mountpoint for Amazon S3.
- Documentation: https://docs.rs/mountpoint-s3-crt-sys/
- License: Apache-2.0
-
Latest release: 0.15.0
published 7 months ago
Rankings
Maintainers (3)
crates.io: mountpoint-s3-crt
Rust interface to the AWS Common Runtime for Mountpoint for Amazon S3.
- Documentation: https://docs.rs/mountpoint-s3-crt/
- License: Apache-2.0
-
Latest release: 0.13.0
published 7 months ago
Rankings
Maintainers (3)
crates.io: mountpoint-s3-client
High-performance Amazon S3 client for Mountpoint for Amazon S3.
- Documentation: https://docs.rs/mountpoint-s3-client/
- License: Apache-2.0
-
Latest release: 0.19.0
published 7 months ago
Rankings
Maintainers (3)
crates.io: mount-s3
Placeholder crate for Mountpoint for Amazon S3.
- Homepage: https://github.com/awslabs/mountpoint-s3
- Documentation: https://docs.rs/mount-s3/
- License: Apache-2.0
-
Latest release: 0.0.1
published almost 3 years ago
Rankings
Maintainers (2)
crates.io: mountpoint-s3-fs
Mountpoint S3 main library
- Documentation: https://docs.rs/mountpoint-s3-fs/
- License: Apache-2.0
-
Latest release: 0.7.0
published 7 months ago
Rankings
Maintainers (5)
crates.io: mountpoint-s3-fuser
A fork of fuser - Filesystem in Userspace (FUSE) for Rust - only for use in Mountpoint for Amazon S3
- Homepage: https://github.com/awslabs/mountpoint-s3/tree/fuser/fork
- Documentation: https://docs.rs/mountpoint-s3-fuser/
- License: MIT
-
Latest release: 0.1.0
published 12 months ago
Rankings
Maintainers (1)
crates.io: mountpoint-s3
Placeholder crate for Mountpoint for Amazon S3.
- Homepage: https://github.com/awslabs/mountpoint-s3
- Documentation: https://docs.rs/mountpoint-s3/
- License: Apache-2.0
-
Latest release: 0.0.1
published almost 3 years ago
Rankings
Maintainers (3)
Dependencies
- ./.github/actions/install-dependencies * composite
- actions-rs/toolchain v1 composite
- actions/cache/restore v3 composite
- actions/cache/save v3 composite
- actions/checkout v3 composite
- aws-actions/configure-aws-credentials v3 composite
- benchmark-action/github-action-benchmark v1 composite
- ./.github/actions/install-dependencies * composite
- actions-rs/toolchain v1 composite
- actions/cache/restore v3 composite
- actions/cache/save v3 composite
- actions/checkout v3 composite
- aws-actions/configure-aws-credentials v3 composite
- ./.github/actions/install-dependencies * composite
- actions/checkout v3 composite
- actions/upload-artifact v3 composite
- docker/build-push-action v4 composite
- actions/checkout v3 composite
- taiki-e/create-gh-release-action v1 composite
- ./.github/actions/install-dependencies * composite
- EmbarkStudios/cargo-deny-action v1 composite
- actions-rs/toolchain v1 composite
- actions/cache v3 composite
- actions/checkout v3 composite
- 379 dependencies
- assert_cmd 2.0.6 development
- assert_fs 1.0.9 development
- aws-config 0.56.0 development
- aws-sdk-s3 0.29.0 development
- aws-sdk-sts 0.29.0 development
- base16ct 0.1.1 development
- ctor 0.1.23 development
- filetime 0.2.21 development
- predicates 2.1.2 development
- proptest 1.0.0 development
- proptest-derive 0.3.0 development
- rand 0.8.5 development
- rand_chacha 0.3.1 development
- sha2 0.10.6 development
- shuttle 0.5.0 development
- tempfile 3.4.0 development
- test-case 2.2.2 development
- tokio 1.24.2 development
- walkdir 2.3.3 development
- anyhow 1.0.64
- async-channel 1.8.0
- async-lock 2.6.0
- bytes 1.2.1
- clap 4.1.9
- const_format 0.2.30
- crc32c 0.6.3
- ctrlc 3.2.3
- dashmap 5.5.0
- fuser 0.12.0
- futures 0.3.24
- hdrhistogram 7.5.2
- lazy_static 1.4.0
- libc 0.2.126
- metrics 0.20.1
- mountpoint-s3-client 0.3.0
- mountpoint-s3-crt 0.2.2
- nix 0.26.2
- once_cell 1.16.0
- regex 1.7.1
- serde_json 1.0.95
- supports-color 2.0.0
- syslog 6.1.0
- thiserror 1.0.34
- time 0.3.17
- tracing 0.1.35
- tracing-log 0.1.3
- tracing-subscriber 0.3.14
- anyhow 1.0.64 development
- aws-config 0.56.0 development
- aws-credential-types 0.56.0 development
- aws-sdk-s3 0.29.0 development
- aws-sdk-sts 0.29.0 development
- aws-smithy-runtime-api 0.56.1 development
- bytes 1.2.1 development
- clap 4.1.9 development
- ctor 0.1.23 development
- proptest 1.0.0 development
- rand 0.8.5 development
- rand_chacha 0.3.1 development
- rusty-fork 0.3.0 development
- tempfile 3.5.0 development
- test-case 2.2.2 development
- tokio 1.24.2 development
- tracing-subscriber 0.3.14 development
- async-trait 0.1.57
- auto_impl 1.0.1
- base64ct 1.6.0
- const_format 0.2.30
- futures 0.3.24
- lazy_static 1.4.0
- libc 0.2.126
- libc-stdhandle 0.1.0
- md-5 0.10.5
- metrics 0.20.1
- mountpoint-s3-crt 0.2.2
- percent-encoding 2.2.0
- pin-project 1.0.12
- regex 1.7.1
- serde_json 1.0.104
- static_assertions 1.1.0
- thiserror 1.0.34
- time 0.3.17
- tracing 0.1.35
- xmltree 0.10.3
- criterion 0.5.1 development
- ctor 0.1.23 development
- futures-timer 3.0.2 development
- rand 0.8.5 development
- tempfile 3.4.0 development
- test-case 2.2.2 development
- tracing-subscriber 0.3.14 development
- async-channel 1.8.0
- futures 0.3.24
- libc 0.2.132
- log 0.4.17
- mountpoint-s3-crt-sys 0.2.2
- smallstr 0.3.0
- static_assertions 1.1.0
- thiserror 1.0.35
- amazonlinux 2023 build
- public.ecr.aws/amazonlinux/amazonlinux 2023 build
- public.ecr.aws/docker/library/centos 7 build
- Pillow *
- boto3 *
- certifi *
- lightning >=2.0.6
- s3fs *
- torch >=2.0.1
- torchdata >=0.6.1
- torchvision *
- webdataset *
- actions-rs/toolchain v1 composite
- actions/cache v3 composite
- actions/checkout v3 composite