Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/agnostiqhq/covalent-aws-plugins

Executor plugins interfacing Covalent with various AWS compute platforms

https://github.com/agnostiqhq/covalent-aws-plugins

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ✓
    DOI references
    Found 2 DOI reference(s) in README
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (15.4%) to scientific vocabulary

Keywords

aws aws-batch aws-braket aws-ec2 aws-ecs aws-lambda braket covalent etl hybrid-cloud python quantum

Keywords from Contributors

data-pipeline parallelization pipelines quantum-computing workflow-automation distributed-computing quantum-machine-learning cloud-computing hpc-applications machinelearning-python
Last synced: 5 months ago · JSON representation

Repository

Executor plugins interfacing Covalent with various AWS compute platforms

Basic Info
  • Host: GitHub
  • Owner: AgnostiqHQ
  • License: apache-2.0
  • Language: Python
  • Default Branch: develop
  • Homepage: https://covalent.xyz
  • Size: 499 KB
Statistics
  • Stars: 17
  • Watchers: 11
  • Forks: 0
  • Open Issues: 6
  • Releases: 25
Topics
aws aws-batch aws-braket aws-ec2 aws-ecs aws-lambda braket covalent etl hybrid-cloud python quantum
Created over 3 years ago · Last pushed about 2 years ago
Metadata Files
Readme Changelog License

README.md

 

[![covalent](https://img.shields.io/badge/covalent-0.177.0-purple)](https://github.com/AgnostiqHQ/covalent) [![apache](https://img.shields.io/badge/License-Apache_License_2.0-blue)](https://www.apache.org/licenses/LICENSE-2.0)

Covalent AWS Plugins

Covalent is a python based workflow orchestration tool used to execute HPC and quantum tasks in heterogenous environments.

By installing Covalent AWS Plugins users can leverage a broad plugin ecosystem to execute tasks using AWS resources best fit for each task.

Covalent AWS Plugins installs a set of executor plugins that allow tasks to be run in an EC2 instance, AWS Lambda, AWS ECS Cluster, AWS Batch Compute Environment, and as an AWS Braket Job for tasks requiring Quantum devices.

If you're new to covalent visit our Getting Started Guide.

Installation

To use the AWS plugin ecosystem with Covalent, simply install it with pip:

bash pip install "covalent-aws-plugins[all]"

⚠️You must include [all] in order to install all of the AWS plugins

This will ensure that all the AWS executor plugins listed below are installed.

You may require Terraform to be installed to use the AWS EC2 plugin.

Please read our read the docs (RTD) guide for more detailed information about using AWS plugins.

Included Plugins

While each plugin can be installed separately, covalent-aws-plugins pip package will automatically download all of the plugins listed below.

| | Plugin Name | Use Case | |---|-------------|-------------| |AWS Batch| AWS Batch Executor |Useful for heavy compute workloads (high CPU/memory). Tasks are queued to execute in the user defined Batch compute environment.| |AWS EC2|AWS EC2 Executor|General purpose compute workloads where users can select compute resources. An EC2 instance is auto-provisioned using terraform with selected compute settings to execute tasks.| |AWS Braket|AWS Braket Executor|Suitable for Quantum/Classical hybrid workflows. Tasks are executed using a combination of classical and quantum devices.| |AWS ECS|AWS ECS Executor|Useful for moderate to heavy workloads (low memory requirements). Tasks are executed in an AWS ECS cluster as containers.| |AWS Lambda|AWS Lambda Executor|Suitable for short lived tasks that can be parallalized (low memory requirements). Tasks are executed in serverless AWS Lambda functions.|

Usage

Firstly, import covalent.

python import covalent as ct Secondly, define your executor(s) (expand any of the below plugins).

AWS Batch Executor Read more about how to use this executor in our [docs](https://covalent.readthedocs.io/en/latest/api/executors/awsbatch.html#usage-example). Below are the basics of how it can be used. ```python executor = ct.executor.AWSBatchExecutor( s3_bucket_name = "covalent-batch-qa-job-resources", batch_job_definition_name = "covalent-batch-qa-job-definition", batch_queue = "covalent-batch-qa-queue", batch_execution_role_name = "ecsTaskExecutionRole", batch_job_role_name = "covalent-batch-qa-job-role", batch_job_log_group_name = "covalent-batch-qa-log-group", vcpu = 2, # Number of vCPUs to allocate memory = 3.75, # Memory in GB to allocate time_limit = 300, # Time limit of job in seconds ) ```
AWS EC2 Executor Read more about how to use this executor in our [docs](https://covalent.readthedocs.io/en/latest/api/executors/awsec2.html#usage-example). Below are the basics of how it can be used. ```python executor = ct.executor.EC2Executor( instance_type="t2.micro", volume_size=8, #GiB ssh_key_file="~/.ssh/ec2_key" ) ```
AWS Braket Executor Read more about how to use this executor in our [docs](https://covalent.readthedocs.io/en/latest/api/executors/awsbraket.html#usage-example). Below are the basics of how it can be used. ```python executor = ct.executor.BraketExecutor( s3_bucket_name="braket_s3_bucket", ecr_repo_name="braket_ecr_repo", braket_job_execution_role_name="covalent-braket-iam-role", quantum_device="arn:aws:braket:::device/quantum-simulator/amazon/sv1", classical_device="ml.m5.large", storage=30, ) ```
AWS ECS Executor Read more about how to use this executor in our [docs](https://covalent.readthedocs.io/en/latest/api/executors/awsecs.html#usage-example). Below are the basics of how it can be used. ```python executor = ct.executor.ECSExecutor( s3_bucket_name="covalent-fargate-task-resources", ecr_repo_name="covalent-fargate-task-images", ecs_cluster_name="covalent-fargate-cluster", ecs_task_family_name="covalent-fargate-tasks", ecs_task_execution_role_name="ecsTaskExecutionRole", ecs_task_role_name="CovalentFargateTaskRole", ecs_task_subnet_id="subnet-000000e0", ecs_task_security_group_id="sg-0000000a", ecs_task_log_group_name="covalent-fargate-task-logs", vcpu=1, memory=2 ) ```
AWS Lambda Executor Read more about how to use this executor in our [docs](https://covalent.readthedocs.io/en/latest/api/executors/awslambda.html#usage-example). Below are the basics of how it can be used. ```python executor = ct.executor.AWSLambdaExecutor( lambda_role_name="CovalentLambdaExecutionRole", s3_bucket_name="covalent-lambda-job-resources", timeout=60, memory_size=512 ) ```
  • Lastly, define a workflow to execute a particular task using one of the above executors

```python @ct.electron( executor=executor ) def compute_pi(n): # Leibniz formula for π return 4 * sum(1.0/(2i + 1)(-1)**i for i in range(n))

@ct.lattice def workflow(n): return compute_pi(n)

dispatchid = ct.dispatch(workflow)(100000000) result = ct.getresult(dispatchid=dispatchid, wait=True) print(result.result)

```

Which should output

3.141592643589326

Release Notes

Release notes are available in the Changelog.

Citation

Please use the following citation in any publications:

W. J. Cunningham, S. K. Radha, F. Hasan, J. Kanem, S. W. Neagle, and S. Sanand. Covalent. Zenodo, 2022. https://doi.org/10.5281/zenodo.5903364

License

Covalent is licensed under the Apache License 2.0. See the LICENSE file or contact the support team for more details.

Owner

  • Name: Agnostiq
  • Login: AgnostiqHQ
  • Kind: organization
  • Email: contact@agnostiq.ai
  • Location: Toronto

Developing Software for Advanced Computing

GitHub Events

Total
Last Year

Committers

Last synced: 10 months ago

All Time
  • Total Commits: 62
  • Total Committers: 8
  • Avg Commits per committer: 7.75
  • Development Distribution Score (DDS): 0.597
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
CovalentOpsBot c****t 25
Alejandro Esquivel ae@a****d 16
Faiyaz Hasan f****1@g****m 9
Venkat Bala 1****a 6
Scott Wyman Neagle w****a@p****m 2
Will Cunningham w****7 2
Sankalp Sanand s****p@a****i 1
Ara Ghukasyan 3****s 1
Committer Domains (Top 20 + Academic)
agnostiq.ai: 1 alejandro.ltd: 1

Issues and Pull Requests

Last synced: 9 months ago

All Time
  • Total issues: 13
  • Total pull requests: 36
  • Average time to close issues: 15 days
  • Average time to close pull requests: 2 days
  • Total issue authors: 5
  • Total pull request authors: 8
  • Average comments per issue: 0.85
  • Average comments per pull request: 0.83
  • Merged pull requests: 33
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • FyzHsn (4)
  • venkatBala (4)
  • AlejandroEsquivel (2)
  • cjao (1)
  • kessler-frost (1)
Pull Request Authors
  • AlejandroEsquivel (16)
  • FyzHsn (9)
  • venkatBala (6)
  • cjao (1)
  • araghukas (1)
  • kessler-frost (1)
  • scottwn (1)
Top Labels
Issue Labels
bug :bug: (3) priority / high (3) Team West (2) Team East (2) Epic (1) docs (1) infra (1) testing / validation (1)
Pull Request Labels
Team West (1)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 6,160 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 28
  • Total maintainers: 1
pypi.org: covalent-aws-plugins

Covalent AWS Plugins

  • Versions: 28
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 6,160 Last month
Rankings
Downloads: 6.3%
Dependent packages count: 6.6%
Average: 19.5%
Stargazers count: 23.3%
Forks count: 30.5%
Dependent repos count: 30.6%
Maintainers (1)
wjcunningham7
Last synced: 6 months ago

Dependencies

.github/workflows/changelog.yml actions
  • EndBug/add-and-commit v9 composite
  • actions/checkout v3 composite
.github/workflows/changelog_reminder.yml actions
  • actions/checkout master composite
  • peterjgrainger/action-changelog-reminder v1.3.0 composite
.github/workflows/docker.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v4 composite
  • aws-actions/configure-aws-credentials v1 composite
  • docker/build-push-action v2 composite
  • docker/setup-buildx-action v2 composite
  • docker/setup-qemu-action v2 composite
.github/workflows/license.yml actions
  • actions/checkout v3 composite
  • pilosus/action-pip-license-checker main composite
.github/workflows/release.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
  • ncipollo/release-action v1 composite
.github/workflows/tests.yml actions
  • actions-ecosystem/action-get-latest-tag v1 composite
  • actions/checkout v3 composite
  • actions/setup-python v2 composite
  • codecov/codecov-action v3 composite
.github/workflows/version.yml actions
  • AgnostiqHQ/covalent/.github/actions/version develop composite
  • actions/checkout v3 composite
Dockerfile docker
  • ${COVALENT_BASE_IMAGE} latest build
setup.py pypi
requirements.txt pypi
  • boto3 ==1.24.35
  • covalent ==0.177.0.post1.dev0
tests/requirements.txt pypi
  • pytest ==6.2.5 test
  • pytest-cov ==2.12.0 test
  • pytest-mock ==3.6.1 test