Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/awslabs/aws-c-s3

C99 library implementation for communicating with the S3 service, designed for maximizing throughput on high bandwidth EC2 instances.

https://github.com/awslabs/aws-c-s3

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (9.0%) to scientific vocabulary

Keywords

aws-sdk hacktoberfest

Keywords from Contributors

diagram labels interaction
Last synced: 9 months ago · JSON representation

Repository

C99 library implementation for communicating with the S3 service, designed for maximizing throughput on high bandwidth EC2 instances.

Basic Info
  • Host: GitHub
  • Owner: awslabs
  • License: apache-2.0
  • Language: C
  • Default Branch: main
  • Homepage:
  • Size: 3.8 MB
Statistics
  • Stars: 143
  • Watchers: 23
  • Forks: 46
  • Open Issues: 15
  • Releases: 146
Topics
aws-sdk hacktoberfest
Created almost 6 years ago · Last pushed 9 months ago
Metadata Files
Readme Contributing License Code of conduct Notice

README.md

AWS C S3

The AWS-C-S3 library is an asynchronous AWS S3 client focused on maximizing throughput and network utilization.

Key features:

  • Automatic Request Splitting: Improves throughput by automatically splitting the request into part-sized chunks and performing parallel uploads/downloads of these chunks over multiple connections. There's a cap on the throughput of single S3 connection, the only way to go faster is multiple parallel connections.
  • Automatic Retries: Increases resilience by retrying individual failed chunks of a file transfer, eliminating the need to restart transfers from scratch after an intermittent error.
  • DNS Load Balancing: DNS resolver continuously harvests Amazon S3 IP addresses. When load is spread across the S3 fleet, overall throughput more reliable than if all connections are going to a single IP.
  • Advanced Network Management: The client incorporates automatic request parallelization, effective timeouts and retries, and efficient connection reuse. This approach helps to maximize throughput and network utilization, and to avoid network overloads.
  • Thread Pools and Async I/O: Avoids bottlenecks associated with single-thread processing.
  • Parallel Reads: When uploading a large file from disk, reads from multiple parts of the file in parallel. This is faster than reading the file sequentially from beginning to end.

Documentation

License

This library is licensed under the Apache 2.0 License.

Usage

Building

CMake 3.9+ is required to build.

<install-path> must be an absolute path in the following instructions.

Linux-Only Dependencies

If you are building on Linux, you will need to build aws-lc and s2n-tls first.

``` git clone git@github.com:aws/aws-lc.git cmake -S aws-lc -B aws-lc/build -DCMAKEINSTALLPREFIX= cmake --build aws-lc/build --target install

git clone git@github.com:aws/s2n-tls.git cmake -S s2n-tls -B s2n-tls/build -DCMAKEINSTALLPREFIX= -DCMAKEPREFIXPATH= cmake --build s2n-tls/build --target install ```

Building aws-c-s3 and Remaining Dependencies

``` git clone git@github.com:awslabs/aws-c-common.git cmake -S aws-c-common -B aws-c-common/build -DCMAKEINSTALLPREFIX= cmake --build aws-c-common/build --target install

git clone git@github.com:awslabs/aws-checksums.git cmake -S aws-checksums -B aws-checksums/build -DCMAKEINSTALLPREFIX= -DCMAKEPREFIXPATH= cmake --build aws-checksums/build --target install

git clone git@github.com:awslabs/aws-c-cal.git cmake -S aws-c-cal -B aws-c-cal/build -DCMAKEINSTALLPREFIX= -DCMAKEPREFIXPATH= cmake --build aws-c-cal/build --target install

git clone git@github.com:awslabs/aws-c-io.git cmake -S aws-c-io -B aws-c-io/build -DCMAKEINSTALLPREFIX= -DCMAKEPREFIXPATH= cmake --build aws-c-io/build --target install

git clone git@github.com:awslabs/aws-c-compression.git cmake -S aws-c-compression -B aws-c-compression/build -DCMAKEINSTALLPREFIX= -DCMAKEPREFIXPATH= cmake --build aws-c-compression/build --target install

git clone git@github.com:awslabs/aws-c-http.git cmake -S aws-c-http -B aws-c-http/build -DCMAKEINSTALLPREFIX= -DCMAKEPREFIXPATH= cmake --build aws-c-http/build --target install

git clone git@github.com:awslabs/aws-c-sdkutils.git cmake -S aws-c-sdkutils -B aws-c-sdkutils/build -DCMAKEINSTALLPREFIX= -DCMAKEPREFIXPATH= cmake --build aws-c-sdkutils/build --target install

git clone git@github.com:awslabs/aws-c-auth.git cmake -S aws-c-auth -B aws-c-auth/build -DCMAKEINSTALLPREFIX= -DCMAKEPREFIXPATH= cmake --build aws-c-auth/build --target install

git clone git@github.com:awslabs/aws-c-s3.git cmake -S aws-c-s3 -B aws-c-s3/build -DCMAKEINSTALLPREFIX= -DCMAKEPREFIXPATH= cmake --build aws-c-s3/build --target install ```

Running S3 sample

After installing all the dependencies, and building aws-c-s3, you can run the sample directly from the s3 build directory.

To download: aws-c-s3/build/samples/s3/s3 cp s3://<bucket-name>/<object-name> <download-path> --region <region> To upload: aws-c-s3/build/samples/s3/s3 cp <upload-path> s3://<bucket-name>/<object-name> --region <region> To list objects: aws-c-s3/build/samples/s3/s3 ls s3://<bucket-name> --region <region>

Testing

The unit tests require an AWS account with S3 buckets set up in a particular way. Use the test_helper script to set this up.

Owner

  • Name: Amazon Web Services - Labs
  • Login: awslabs
  • Kind: organization
  • Location: Seattle, WA

AWS Labs

Committers

Last synced: about 3 years ago

All Time
  • Total Commits: 227
  • Total Committers: 22
  • Avg Commits per committer: 10.318
  • Development Distribution Score (DDS): 0.709
Past Year
  • Commits: 63
  • Committers: 10
  • Avg Commits per committer: 6.3
  • Development Distribution Score (DDS): 0.556
Top Committers
Name Email Commits
Ryan Carper 5****r@u****m 66
Dengke Tang d****t@a****m 30
Ryan Carper r****r@a****m 23
Michael Graeb g****m@a****m 18
Dengke Tang 8****5@q****m 17
Justin Boswell b****j@a****m 15
Bret Ambrose b****e@g****m 11
ilevyor 8****r@u****m 7
Cesar Mello m****r@a****m 7
Dmitriy Musatkin 6****n@u****m 7
Gerrit Renker g****r@u****m 5
Waqar Ahmed Khan w****7@g****m 5
Jonathan M. Henson h****o@a****m 3
Cesar Mello c****o@g****m 2
Joseph Klix j****x@g****m 2
Pushen Wang w****0@h****m 2
dependabot[bot] 4****]@u****m 2
Sergey Ryabinin s****n@a****m 1
Amazon GitHub Automation 5****o@u****m 1
Eric Ace e****e@p****m 1
Steve Kim 8****K@u****m 1
TwistedTwigleg t****g@g****m 1
Committer Domains (Top 20 + Academic)
amazon.com: 7 qq.com: 1

Issues and Pull Requests

Last synced: 9 months ago

All Time
  • Total issues: 29
  • Total pull requests: 435
  • Average time to close issues: 3 months
  • Average time to close pull requests: 22 days
  • Total issue authors: 17
  • Total pull request authors: 26
  • Average comments per issue: 3.55
  • Average comments per pull request: 0.84
  • Merged pull requests: 296
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 6
  • Pull requests: 156
  • Average time to close issues: 6 days
  • Average time to close pull requests: 4 days
  • Issue authors: 3
  • Pull request authors: 10
  • Average comments per issue: 2.33
  • Average comments per pull request: 0.81
  • Merged pull requests: 91
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • grrtrr (11)
  • djchapm (2)
  • dbickson (2)
  • EUtrilla2002 (1)
  • nunotexbsd (1)
  • DmitriyMusatkin (1)
  • snickl (1)
  • bpiper (1)
  • jmklix (1)
  • Bbulatov (1)
  • changpe1 (1)
  • alexsmith1612 (1)
  • graebm (1)
  • joeshaw (1)
  • mattBrzezinski (1)
Pull Request Authors
  • waahm7 (159)
  • TingDaoK (129)
  • graebm (55)
  • DmitriyMusatkin (39)
  • grrtrr (18)
  • bretambrose (4)
  • jmklix (4)
  • sbSteveK (3)
  • vladem (2)
  • ashishdhingra (2)
  • rccarper (2)
  • thundergolfer (2)
  • ProgramComputer (2)
  • jamesbornholt (2)
  • israel4435-ux (1)
Top Labels
Issue Labels
bug (12) p2 (8) feature-request (5) needs-triage (5) documentation (2) p3 (1) p1 (1) potential-regression (1)
Pull Request Labels

Packages

  • Total packages: 15
  • Total downloads:
    • homebrew 6,130 last-month
  • Total dependent packages: 9
    (may contain duplicates)
  • Total dependent repositories: 23
    (may contain duplicates)
  • Total versions: 177
  • Total maintainers: 3
alpine-edge: aws-c-s3-dev

AWS C99 library implementation for communicating with the S3 service (development files)

  • Versions: 46
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 14.6%
Average: 15.0%
Forks count: 21.6%
Stargazers count: 23.8%
Maintainers (1)
achill@achill.org
Last synced: 10 months ago
alpine-edge: aws-c-s3

AWS C99 library implementation for communicating with the S3 service

  • Versions: 46
  • Dependent Packages: 1
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 14.6%
Average: 15.0%
Forks count: 21.6%
Stargazers count: 23.8%
Maintainers (1)
achill@achill.org
Last synced: 9 months ago
conda-forge.org: aws-c-s3
  • Versions: 31
  • Dependent Packages: 2
  • Dependent Repositories: 23
Rankings
Dependent repos count: 7.6%
Dependent packages count: 19.6%
Average: 25.6%
Forks count: 34.2%
Stargazers count: 41.0%
Last synced: 9 months ago
formulae.brew.sh: aws-c-s3

C99 library implementation for communicating with the S3 service

  • Versions: 22
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 6,130 Last month
Rankings
Dependent packages count: 17.9%
Average: 45.5%
Dependent repos count: 52.0%
Downloads: 66.5%
Last synced: 9 months ago
anaconda.org: aws-c-s3

The AWS-C-S3 library is an asynchronous AWS S3 client focused on maximizing throughput and network utilization.

  • Versions: 4
  • Dependent Packages: 2
  • Dependent Repositories: 0
Rankings
Dependent packages count: 50.7%
Average: 55.2%
Dependent repos count: 59.6%
Last synced: 9 months ago
alpine-v3.18: aws-c-s3-dev

AWS C99 library implementation for communicating with the S3 service (development files)

  • Versions: 9
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
alice@ayaya.dev
Last synced: 9 months ago
alpine-v3.22: aws-c-s3

AWS C99 library implementation for communicating with the S3 service

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
fossdd@pwned.life
Last synced: 9 months ago
alpine-v3.20: aws-c-s3-dev

AWS C99 library implementation for communicating with the S3 service (development files)

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Last synced: 9 months ago
alpine-v3.20: aws-c-s3

AWS C99 library implementation for communicating with the S3 service

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Last synced: 9 months ago
alpine-v3.19: aws-c-s3

AWS C99 library implementation for communicating with the S3 service

  • Versions: 2
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Last synced: 9 months ago
alpine-v3.22: aws-c-s3-dev

AWS C99 library implementation for communicating with the S3 service (development files)

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
fossdd@pwned.life
Last synced: 9 months ago
alpine-v3.19: aws-c-s3-dev

AWS C99 library implementation for communicating with the S3 service (development files)

  • Versions: 2
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Last synced: 9 months ago
alpine-v3.21: aws-c-s3

AWS C99 library implementation for communicating with the S3 service

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
fossdd@pwned.life
Last synced: 9 months ago
alpine-v3.21: aws-c-s3-dev

AWS C99 library implementation for communicating with the S3 service (development files)

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
fossdd@pwned.life
Last synced: 9 months ago
alpine-v3.18: aws-c-s3

AWS C99 library implementation for communicating with the S3 service

  • Versions: 9
  • Dependent Packages: 4
  • Dependent Repositories: 0
Rankings
Dependent repos count: 0.0%
Dependent packages count: 0.0%
Average: 100%
Maintainers (1)
alice@ayaya.dev
Last synced: 9 months ago

Dependencies

.github/workflows/ci.yml actions
  • actions/checkout v3 composite
.github/workflows/clang-format.yml actions
  • DoozyX/clang-format-lint-action v0.3.1 composite
  • actions/checkout v1 composite
.github/workflows/closed-issue-message.yml actions
  • aws-actions/closed-issue-message v1 composite
.github/workflows/codecov.yml actions
  • actions/checkout v3 composite
.github/workflows/handle-stale-discussions.yml actions
  • aws-github-ops/handle-stale-discussions v1 composite
.github/workflows/stale_issue.yml actions
  • aws-actions/stale-issue-cleanup v3 composite
benchmarks/benchmarks-stack/benchmarks-stack/package.json npm
  • @aws-cdk/assert ^v1.126.0 development
  • @types/jest ^27.0.1 development
  • @types/node 10.17.27 development
  • aws-cdk ^1.20.0 development
  • jest ^27.2.1 development
  • ts-jest ^27.0.5 development
  • ts-node ^10.9.1 development
  • typescript ^4.2.0 development
  • @aws-cdk/aws-ec2 ^v1.126.0
  • @aws-cdk/aws-iam ^v1.126.0
  • @aws-cdk/aws-lambda ^v1.126.0
  • @aws-cdk/aws-s3 ^v1.126.0
  • @aws-cdk/aws-s3-assets ^v1.126.0
  • @aws-cdk/core ^v1.126.0
  • aws-sdk ^2.910.0
  • source-map-support ^0.5.16
benchmarks/dashboard-stack/package.json npm
  • @aws-cdk/assert ^v1.126.0 development
  • @types/jest ^27.0.1 development
  • @types/node 10.17.27 development
  • aws-cdk ^1.20.0 development
  • jest ^27.2.1 development
  • ts-jest ^27.0.5 development
  • ts-node ^10.9.1 development
  • typescript ^4.2.0 development
  • @aws-cdk/aws-cloudwatch ^v1.126.0
  • @aws-cdk/aws-codebuild ^v1.126.0
  • @aws-cdk/aws-ec2 ^v1.126.0
  • @aws-cdk/aws-events ^v1.126.0
  • @aws-cdk/aws-events-targets ^v1.126.0
  • @aws-cdk/aws-iam ^v1.126.0
  • @aws-cdk/aws-kms ^v1.126.0
  • @aws-cdk/aws-lambda ^v1.126.0
  • @aws-cdk/aws-s3 ^v1.126.0
  • @aws-cdk/aws-s3-assets ^v1.126.0
  • @aws-cdk/aws-s3-deployment ^v1.126.0
  • @aws-cdk/core ^v1.126.0
  • cdk-ec2-key-pair 2.2.0
  • source-map-support ^0.5.16