Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/asfhyp3/s2-staging

Small app for copying Sentinel-2 data from Google Cloud Storage to AWS S3

https://github.com/asfhyp3/s2-staging

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (6.8%) to scientific vocabulary
Last synced: 10 months ago · JSON representation

Repository

Small app for copying Sentinel-2 data from Google Cloud Storage to AWS S3

Basic Info
  • Host: GitHub
  • Owner: ASFHyP3
  • License: bsd-3-clause
  • Language: Python
  • Default Branch: main
  • Size: 14.6 KB
Statistics
  • Stars: 0
  • Watchers: 3
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Archived
Created over 1 year ago · Last pushed over 1 year ago
Metadata Files
Readme License

README.md

s2-staging

Small app for copying Sentinel-2 B08.jp2 files from Google Cloud Storage to AWS S3

For example, for scene S2A_MSIL1C_20231213T235751_N0510_R087_T51CWT_20231214T003659: - https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/51/C/WT/S2AMSIL1C20231213T235751N0510R087T51CWT20231214T003659.SAFE/./GRANULE/L1CT51CWTA04427120231213T235749/IMGDATA/T51CWT20231213T235751B08.jp2

is copied to: - s3://myBucket/myPrefix/S2A_MSIL1C_20231213T235751_N0510_R087_T51CWT_20231214T003659_B08.jp2

Local Usage

Requires AWS credentials with s3:PutObject permissions for arn:aws:s3:::myBucket/myPrefix/*.

commandline git clone git@github.com:ASFHyP3/s2-staging.git cd s2-staging mamba env create -f environment.yml mamba activate s2-staging cd src python

python from main import fetch_scene fetch_scene('S2B_MSIL1C_20240216T235509_N0510_R087_T51CWT_20240217T005730', 'myBucket', 'myPrefix/')

AWS Usage

Requires AWS credentials with permission to deploy the CloudFormation stack into JPL-managed AWS.

``` git clone git@github.com:ASFHyP3/s2-staging.git cd s2-staging mamba env create -f environment.yml mamba activate s2-staging

python -m pip install -r requirements.txt -t src/ aws cloudformation package \ --template-file cloudformation.yml \ --s3-bucket myBucket \ --s3-prefix myPrefix \ --output-template-file packaged.yml aws cloudformation deploy \ --template-file packaged.yml \ --stack-name s2-staging \ --capabilities CAPABILITY_IAM \ --parameter-overrides BucketName=its-live-project BucketPrefix=s2-cache/

aws sqs send-message \ --queue-url https://sqs.us-west-2.amazonaws.com/123456789012/s2-staging-Queue-MxuL1mLZVvUO \ --message-body S2AMSIL1C20231213T235751N0510R087T51CWT20231214T003659 ```

See batchsqssubmit.py for an example of submitting many scenes using sqs.sendmessagebatch.

Owner

  • Name: HyP3
  • Login: ASFHyP3
  • Kind: organization
  • Location: Fairbanks, AK

Alaska Satellite Facility's Hybrid Pluggable Processing Pipeline

GitHub Events

Total
  • Delete event: 4
  • Push event: 5
  • Pull request review event: 4
  • Pull request event: 8
  • Create event: 4
Last Year
  • Delete event: 4
  • Push event: 5
  • Pull request review event: 4
  • Pull request event: 8
  • Create event: 4

Dependencies

requirements.txt pypi
  • boto3 ==1.35.40
  • requests ==2.32.3
environment.yml conda
  • pip
  • python 3.12.*