aws-parallelcluster
AWS ParallelCluster is an AWS supported Open Source cluster management tool to deploy and manage HPC clusters in the AWS cloud.
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
2 of 119 committers (1.7%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (15.7%) to scientific vocabulary
Keywords from Contributors
Repository
AWS ParallelCluster is an AWS supported Open Source cluster management tool to deploy and manage HPC clusters in the AWS cloud.
Basic Info
- Host: GitHub
- Owner: aws
- License: apache-2.0
- Language: Python
- Default Branch: develop
- Homepage: https://github.com/aws/aws-parallelcluster
- Size: 26.8 MB
Statistics
- Stars: 873
- Watchers: 86
- Forks: 313
- Open Issues: 178
- Releases: 95
Metadata Files
README.md
AWS ParallelCluster - HPC for the Cloud
AWS ParallelCluster is an AWS supported Open Source cluster management tool that makes it easy for you to deploy and manage High Performance Computing (HPC) clusters in the AWS cloud. Built on the Open Source CfnCluster project, AWS ParallelCluster enables you to quickly build an HPC compute environment in AWS. It automatically sets up the required compute resources and a shared filesystem and offers a variety of batch schedulers such as AWS Batch and Slurm. AWS ParallelCluster facilitates both quick start proof of concepts (POCs) and production deployments. You can build higher level workflows, such as a Genomics portal that automates the entire DNA sequencing workflow, on top of AWS ParallelCluster.
Quick Start
IMPORTANT: you will need an Amazon EC2 Key Pair to be able to complete the following steps. Please see the Official AWS Guide.
First, prepare a Python Virtual Environment for ParallelCluster, note ParallelCluster >= 3.0.0 requires Python >= 3.7.
python3 -m pip install --upgrade pip
python3 -m pip install --user --upgrade virtualenv
python3 -m virtualenv ~/hpc-ve
source ~/hpc-ve/bin/activate
Make sure you have installed the AWS Command Line Interface:
$ pip3 install awscli
Node.js is required by AWS CDK library used by ParallelCluster. Please reference installation instructions in the AWS CDK documentaton.
Then you can install AWS ParallelCluster:
$ pip3 install aws-parallelcluster
Next, configure your aws credentials and default region:
$ aws configure
AWS Access Key ID [None]: YOUR_KEY
AWS Secret Access Key [None]: YOUR_SECRET
Default region name [us-east-1]:
Default output format [None]:
Then, run pcluster configure. A list of valid options will be displayed for each
configuration parameter. Type an option number and press Enter to select a specific option,
or just press Enter to accept the default option.
``` $ pcluster configure --config /dir/cluster-config.yaml INFO: Configuration file /dir/cluster-config.yaml will be written. Press CTRL-C to interrupt the procedure.
Allowed values for AWS Region ID: 1. eu-north-1 ... 15. us-west-1 16. us-west-2 AWS Region ID [us-east-1]: ```
Be sure to select a region containing the EC2 key pair you wish to use. You can also import a public key using these instructions.
During the process you will be asked to set up your networking environment. The wizard will offer you the choice of using an existing VPC or creating a new one on the fly.
Automate VPC creation? (y/n) [n]:
Enter n if you already have a VPC suitable for the cluster. Otherwise you can let pcluster configure
create a VPC for you. The same choice is given for the subnet: you can select a valid subnet ID for
both the head node and compute nodes, or you can let pcluster configure set up everything for you.
The same choice is given for the subnet configuration: you can select a valid subnet ID for both
the head node and compute nodes, or you can let pcluster configure set up everything for you.
In the latter case, just select the configuration you prefer.
Automate Subnet creation? (y/n) [y]: y
Allowed values for Network Configuration:
1. Head node in a public subnet and compute fleet in a private subnet
2. Head node and compute fleet in the same public subnet
At the end of the process a message like this one will be shown:
Configuration file written to /dir/conf_file
You can edit your configuration file or simply run 'pcluster create-cluster --cluster-name cluster-name --cluster-configuration /dir/cluster-config.yaml' to create your cluster.
Now you can create your first cluster:
$ pcluster create-cluster --cluster-name myfirstcluster --cluster-configuration /dir/cluster-config.yaml
After the cluster finishes creating, log in:
$ pcluster ssh --cluster-name myfirstcluster
You can view the running compute hosts:
$ sinfo
For more information on any of these steps see the Getting Started Guide.
Documentation
We've been working hard to greatly improve the Documentation, it's now published in 10 languages, one of the many benefits of being hosted on AWS Docs. Of most interest to new users is the Getting Started Guide.
If you have changes you would like to see in the docs, please either submit feedback using the feedback link at the bottom of each page or create an issue or pull request for the project at: https://github.com/awsdocs/aws-parallelcluster-user-guide.
Issues
Please open a GitHub issue for any feedback or issues: https://github.com/aws/aws-parallelcluster/issues. There is also an active AWS HPC forum which may be helpful: https://repost.aws/tags/TAbl-DsTlyQMe0T2i-d5Rr8g/aws-parallel-cluster.
Changes
CfnCluster to AWS ParallelCluster
In Version 2.0.0, we changed the name of CfnCluster to AWS ParallelCluster. With that name change we released several new features, which you can read about in the Change Log.
Owner
- Name: Amazon Web Services
- Login: aws
- Kind: organization
- Email: open-source-github@amazon.com
- Location: United States of America
- Website: https://amazon.com/aws
- Repositories: 502
- Profile: https://github.com/aws
GitHub Events
Total
- Create event: 21
- Release event: 4
- Issues event: 106
- Watch event: 43
- Delete event: 4
- Issue comment event: 314
- Push event: 331
- Gollum event: 31
- Pull request review comment event: 428
- Pull request review event: 763
- Pull request event: 718
- Fork event: 9
Last Year
- Create event: 21
- Release event: 4
- Issues event: 106
- Watch event: 43
- Delete event: 4
- Issue comment event: 314
- Push event: 331
- Gollum event: 31
- Pull request review comment event: 428
- Pull request review event: 763
- Pull request event: 718
- Fork event: 9
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Francesco De Martino | f****m@a****m | 778 |
| Enrico Usai | u****i@a****m | 747 |
| Hanwen | h****i@a****m | 607 |
| Luca Carrogu | c****u@a****m | 546 |
| Giacomo Marciani | m****o@a****m | 428 |
| chenwany | c****y@a****m | 247 |
| Sean Smith | s****m@a****m | 245 |
| Tim Lane | t****e@a****m | 207 |
| Dougal Ballantyne | d****b@a****m | 158 |
| Rex | s****c@a****m | 146 |
| Yulei Wang | y****n@a****m | 137 |
| ddeidda | d****a@a****m | 131 |
| Ermanno Moser | e****n@a****m | 130 |
| Jacopo De Amicis | j****s@a****t | 105 |
| Himani Anil Deshpande | 7****1 | 102 |
| Nicola Sirena | n****a@a****m | 101 |
| Charles | c****a@a****m | 94 |
| Ryan Anderson | n****y@a****m | 89 |
| hgreebe | 1****e | 83 |
| Eddy Mwiti | e****i@a****m | 79 |
| Francesco Giordano | g****r@a****t | 76 |
| Dougal Ballantyne | d****m | 73 |
| ParallelCluster AMI bot | e****v@a****m | 71 |
| Xuanqi He | 9****8 | 59 |
| Balaji Sridharan | f****j@a****m | 47 |
| Edoardo Antonini | e****n@a****m | 44 |
| Delong Meng | d****g@a****m | 38 |
| Maurizio Melato | m****o@a****m | 36 |
| Himani Deshpande | h****p@a****m | 36 |
| David Pratt | 1****t | 34 |
| and 89 more... | ||
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 10 months ago
All Time
- Total issues: 321
- Total pull requests: 1,986
- Average time to close issues: 7 months
- Average time to close pull requests: 9 days
- Total issue authors: 163
- Total pull request authors: 45
- Average comments per issue: 2.95
- Average comments per pull request: 0.45
- Merged pull requests: 1,663
- Bot issues: 0
- Bot pull requests: 32
Past Year
- Issues: 81
- Pull requests: 799
- Average time to close issues: 16 days
- Average time to close pull requests: 2 days
- Issue authors: 40
- Pull request authors: 13
- Average comments per issue: 1.14
- Average comments per pull request: 0.11
- Merged pull requests: 661
- Bot issues: 0
- Bot pull requests: 14
Top Authors
Issue Authors
- gwolski (18)
- snemir2 (11)
- francisreyes-tfs (9)
- gmarciani (8)
- hanwen-pcluste (7)
- stefan-maxar (7)
- himani2411 (6)
- nyetsche (6)
- enrico-usai (6)
- joehellmersNOAA (5)
- jagga13 (5)
- hehe7318 (5)
- rmarable-flaretx (5)
- Waqiah (5)
- mrgum (5)
Pull Request Authors
- gmarciani (459)
- hanwen-pcluste (269)
- himani2411 (210)
- hgreebe (181)
- hanwen-cluster (175)
- hehe7318 (139)
- dreambeyondorange (110)
- enrico-usai (84)
- jdeamicis (62)
- NSsirena (49)
- lukeseawalker (46)
- EddyMM (36)
- dependabot[bot] (19)
- judysng (18)
- github-actions[bot] (13)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 5
-
Total downloads:
- pypi 97,255 last-month
-
Total dependent packages: 5
(may contain duplicates) -
Total dependent repositories: 16
(may contain duplicates) - Total versions: 229
- Total maintainers: 9
pypi.org: aws-parallelcluster
AWS ParallelCluster is an AWS supported Open Source cluster management tool to deploy and manage HPC clusters in the AWS cloud.
- Homepage: https://github.com/aws/aws-parallelcluster
- Documentation: https://docs.aws.amazon.com/parallelcluster/
- License: Apache License 2.0
-
Latest release: 3.13.2
published about 1 year ago
Rankings
Maintainers (1)
proxy.golang.org: github.com/aws/aws-parallelcluster
- Documentation: https://pkg.go.dev/github.com/aws/aws-parallelcluster#section-documentation
- License: apache-2.0
-
Latest release: v3.13.2+incompatible
published about 1 year ago
Rankings
pypi.org: aws-parallelcluster-awsbatch-cli
AWS ParallelCluster AWS Batch CLI provides a set of commands to manage AWS Batch resources created by ParallelCluster and AWS Batch jobs.
- Homepage: https://github.com/aws/aws-parallelcluster
- Documentation: https://docs.aws.amazon.com/parallelcluster/
- License: Apache License 2.0
-
Latest release: 1.4.0
published about 2 years ago
Rankings
Maintainers (1)
spack.io: aws-parallelcluster
AWS ParallelCluster is an AWS supported Open Source cluster management tool to deploy and manage HPC clusters in the AWS cloud.
- Homepage: https://github.com/aws/aws-parallelcluster
- License: []
-
Latest release: 2.11.9
published over 3 years ago
Rankings
Maintainers (8)
conda-forge.org: aws-parallelcluster
- Homepage: https://github.com/aws/aws-parallelcluster
- License: Apache-2.0
-
Latest release: 2.11.8
published over 3 years ago
Rankings
Dependencies
- software.amazon.smithy:smithy-aws-apigateway-traits 1.22.0 implementation
- software.amazon.smithy:smithy-aws-traits 1.22.0 implementation
- software.amazon.smithy:smithy-linters 1.22.0 implementation
- software.amazon.smithy:smithy-model 1.22.0 implementation
- boto3 >=1.16.14
- python_dateutil >=2.5.3
- setuptools >=21.0.0
- urllib3 >=1.26.6
- pytest-cov >=2.8.1
- boto3 >=1.16.14
- tabulate *
- assertpy *
- pytest *
- pytest-cov *
- pytest-datadir *
- pytest-html *
- pytest-mock *
- pytest-xdist *
- recordclass *
- PyYAML *
- aws-cdk.aws-batch *
- aws-cdk.aws-cloudformation *
- aws-cdk.aws-codebuild *
- aws-cdk.aws-dynamodb *
- aws-cdk.aws-ec2 *
- aws-cdk.aws-efs *
- aws-cdk.aws-events *
- aws-cdk.aws-fsx *
- aws-cdk.aws-iam *
- aws-cdk.aws-imagebuilder *
- aws-cdk.aws-logs *
- aws-cdk.aws-route53 *
- aws-cdk.aws-sqs *
- aws-cdk.aws-ssm *
- aws-cdk.core *
- aws_cdk.aws-cloudwatch *
- aws_cdk.aws-lambda *
- boto3 >=1.16.14
- connexion *
- flask *
- jinja2 *
- jmespath *
- marshmallow *
- tabulate *
- werkzeug *
- assertpy *
- freezegun *
- jinja2 *
- munch *
- pytest *
- pytest-cov *
- pytest-datadir *
- pytest-html *
- pytest-mock *
- pytest-xdist *
- recordclass *
- assertpy *
- cfn-flip *
- cfn-lint *
- jinja2 *
- pytest *
- cfn_flip *
- cfn_flip *
- jinja2 *
- cfn-lint *
- cfn_flip *
- jinja2 *
- PyYAML *
- argparse *
- assertpy *
- boto3 *
- cfn_flip *
- click *
- fabric ==2.6.0
- filelock *
- jinja2 *
- junitparser *
- matplotlib *
- pexpect *
- psutil *
- pyOpenSSL *
- pykwalify *
- pytest *
- pytest-datadir *
- pytest-html *
- pytest-rerunfailures *
- pytest-sugar *
- pytest-xdist *
- requests *
- retrying *
- troposphere *
- untangle *
- danger >= 0
- actions/checkout v2 composite
- dangoslen/changelog-enforcer v1.4.0 composite
- actions/checkout v2 composite
- actions/setup-java v1 composite
- actions/setup-python v2 composite
- actions/upload-artifact master composite
- dependency-check/Dependency-Check_Action main composite
- ludeeus/action-shellcheck master composite
- mikefarah/yq v4.6.3 composite
- aws-actions/closed-issue-message v1 composite
- actions/checkout v2 composite
- github/codeql-action/analyze v1 composite
- github/codeql-action/init v1 composite
- lee-dohm/no-response v0.5.0 composite
- francesco-giordano/gh-pr-content-checker v1.0.0 composite
- public.ecr.aws/lambda/python 3.9 build
- public.ecr.aws/amazonlinux/amazonlinux 2 build