Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/alan-turing-institute/s3toazure

Move Data from an Amazon S3 Bucket using parallel connections

https://github.com/alan-turing-institute/s3toazure

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (6.1%) to scientific vocabulary

Keywords

aws azure blob-storage s3
Last synced: 6 months ago · JSON representation

Repository

Move Data from an Amazon S3 Bucket using parallel connections

Basic Info
  • Host: GitHub
  • Owner: alan-turing-institute
  • Language: Shell
  • Default Branch: master
  • Size: 3.91 KB
Statistics
  • Stars: 1
  • Watchers: 5
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Topics
aws azure blob-storage s3
Created over 8 years ago · Last pushed over 8 years ago
Metadata Files
Readme

README.md

s3ToAzure

This simple tool provides a way of directly transferring data from an Amazon Web Services S3 bucket to Azure Blob Storage.

Required Permissions

The following permissions are required on AWS:

s3:ListBucket for Amazon S3 Bucket Operations, in order to list the files inside of a bucket. s3:GetObject for Amazon S3 Object Operations

Prerequisites

Install the Microsoft Azure command line tools: https://github.com/Azure/azure-cli Install the Amazon Web Services command line tools: http://docs.aws.amazon.com/cli/latest/userguide/installing.html

Configure the AWS CLI by typing aws configure and entering your credentials Login to the Azure CLI by typing az login

Usage

  1. Get a list of files from the desired S3 bucket, replace <S3 FILE LIST> with the desired file name

bash $ aws s3 ls s3://<S3 BUCKET NAME> | tr -s ' ' | cut -d ' ' -f4 > <S3 FILE LIST>

  1. Run the parallel_download.sh script

bash $ bash parallel_download.sh -i <S3 FILE LIST> -w <NUMBER OF PARALLEL CONNECTIONS> -n <AZURE STORAGE NAME> -k <AZURE STORAGE KEY> -c <AZURE CONTAINER NAME> -r <AWS REGION NAME>

Owner

  • Name: The Alan Turing Institute
  • Login: alan-turing-institute
  • Kind: organization
  • Email: info@turing.ac.uk

The UK's national institute for data science and artificial intelligence.

GitHub Events

Total
Last Year

Issues and Pull Requests

Last synced: 12 months ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels