Data

Tools

Indexes

Applications

Experiments

Open Source Science

hpc-rocket

https://github.com/svenmarcus/hpc-rocket

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 3 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.2%) to scientific vocabulary

Keywords

ci continuous-integration hpc
Last synced: 6 months ago · JSON representation ·

Repository

Basic Info
Statistics
  • Stars: 26
  • Watchers: 2
  • Forks: 3
  • Open Issues: 9
  • Releases: 7
Topics
ci continuous-integration hpc
Created almost 5 years ago · Last pushed 8 months ago
Metadata Files
Readme License Citation Authors

README.md

🚀 HPC Rocket

DOI Python application Quality Gate Status Coverage Static Badge

HPC Rocket is a tool to send slurm commands to a remote machine and monitor the job progress. It was primarily written to launch slurm jobs from a CI pipeline.

Installation

You can get the latest version of HPC Rocket on PyPI: python3 -m pip install hpc-rocket

Authentication

HPC Rocket does support authentication via password and private key. Both can be set via environment variables.

Slurm configuration

Currently all sbatch configuration must happen in the job file. HPC Rocket does not offer any other way of configuring your batch jobs.

Configuration file

HPC Rocket uses a configuration file in YAML format containing credentials to connect to the remote machine. Additionally it allows copying files to the remote machine, copying results back to the local machine (collecting) and eventually cleaning up copied or produced files. Note that all paths in the configuration file must be relative paths. On the local machine paths are evaluated from the current working directory, on the remote machine from the user's home directory, unless absolute paths are specified. If you want to overwrite existing files on the remote machine, make sure to specify the overwrite instruction for each file you would like to overwrite. HPC Rocket will evaluate environment variables on the LOCAL machine in the form of ${VAR} and $VAR when parsing the config file.

```yaml host: cluster.example.com user: myuser privatekeyfile: ~/.ssh/idrsa

proxyjumps: - host: myproxy.example.com user: myproxy-user privatekeyfile: ~/.ssh/proxykeyfile

copy: - from: jobs/slurm.job to: slurm.job overwrite: true

  • from: bin/myexecutable to: myexecutable

collect: - from: remoteslurmresult.out to: localslurmresult.out overwrite: true

clean: - slurm.job - myexecutable

sbatch: slurm.job ```

Usage

Launching a job on the remote machine

Use the launch command to launch a job on the remote machine. You must provide a configuration file. The optional --watch flag makes hpc-rocket wait until your job is finished (defaults to false). The collection and cleaning steps in the configuration file are only executed if --watch is set.

bash hpc-rocket launch --watch config.yml

Checking a job's status

If a job was launched without --watch you can still check its status using the status command. You will need to provide a configuration file with connection data and a job ID to check.

bash hpc-rocket status config.yml 12345

Monitoring a job until it finishes

Similar to the status command, hpc-rocket also provides the watch command to monitor a job's status continuously by entering a config file and a job id.

bash hpc-rocket watch config.yml 12345

Canceling a running job

Jobs may also be canceled using the cancel command. Like the previous commands it accepts a config file and the id of a running job.

bash hpc-rocket cancel config.yml 12345

Owner

  • Name: Sven Marcus
  • Login: SvenMarcus
  • Kind: user
  • Location: Braunschweig, Germany
  • Company: @TUBS-Suresoft

Research Software Developer at TU Braunschweig

Citation (CITATION.cff) 

cff-version: 1.2.0
message: "If you use this software, please cite it using these metadata."
abstract: "A commandline tool to run Slurm sbatch commands over SSH"
type: software
license: MIT
authors:
  - family-names: Marcus
    given-names: Sven
    orcid: https://orcid.org/0000-0003-3689-2162
title: hpc-rocket
version: 0.6.3
url: https://github.com/SvenMarcus/hpc-rocket
repository-code: https://github.com/SvenMarcus/hpc-rocket
references:
  - title: rich
    authors:
      - family-names: McGugan
        given-names: Will
    type: software
    license: MIT
    url: https://github.com/willmcgugan/rich

  - title: pyfilesystem2
    authors:
      - family-names: McGugan
        given-names: Will

      - family-names: Larralde
        given-names: Martin

      - family-names: Cimino
        given-names: Giampaolo

      - family-names: Jukes
        given-names: Geoff
    type: software
    license: MIT
    url: https://github.com/PyFilesystem/pyfilesystem2

  - title: fs.sshfs
    authors:
      - family-names: Larralde
        given-names: Martin

      - family-names: Bauer
        given-names: Reimar

      - family-names: Gessinger
        given-names: Paul

      - family-names: Kryński
        given-names: Mariusz

      - family-names: McGugan
        given-names: Will

      - family-names: Nimmer
        given-names: Jeremy

      - family-names: Nyborg
        given-names: Tim

      - family-names: Sutherland
        given-names: Danica J.

      - family-names: Vandenberg
        given-names: John
    type: software
    license: LGPL-2.1
    url: https://github.com/althonos/fs.sshfs

  - title: Paramiko
    authors:
      - family-names: Pointer
        given-names: Robey

      - family-names: Forcier
        given-names: Jeff
    type: software
    license: LGPL-2.1
    url: https://github.com/paramiko/paramiko

  - title: PyYAML
    authors:
      - family-names: Simonov
        given-names: Kirill

    type: software
    license: MIT
    url: https://github.com/yaml/pyyaml

GitHub Events

Total
  • Issues event: 3
  • Watch event: 1
  • Issue comment event: 2
  • Push event: 7
Last Year
  • Issues event: 3
  • Watch event: 1
  • Issue comment event: 2
  • Push event: 7

Committers

Last synced: almost 3 years ago

All Time
  • Total Commits: 283
  • Total Committers: 3
  • Avg Commits per committer: 94.333
  • Development Distribution Score (DDS): 0.049
Top Committers
Name Email Commits
Sven Marcus s****s@o****e 269
Sven Marcus S****s@u****m 13
LGTM Migrator l****r@u****m 1
Committer Domains (Top 20 + Academic)
outlook.de: 1

Issues and Pull Requests

Last synced: 10 months ago

All Time
  • Total issues: 28
  • Total pull requests: 24
  • Average time to close issues: about 2 months
  • Average time to close pull requests: 6 days
  • Total issue authors: 4
  • Total pull request authors: 3
  • Average comments per issue: 0.32
  • Average comments per pull request: 1.5
  • Merged pull requests: 23
  • Bot issues: 0
  • Bot pull requests: 1
Past Year
  • Issues: 2
  • Pull requests: 1
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 2
  • Pull request authors: 1
  • Average comments per issue: 1.5
  • Average comments per pull request: 8.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • SvenMarcus (18)
  • soerenPeters (7)
  • CarMoreno (1)
  • MartinGrignard (1)
Pull Request Authors
  • SvenMarcus (22)
  • ignatiusm (2)
  • lgtm-com[bot] (1)
Top Labels
Issue Labels
enhancement (13) high priority (5) bug (5) low priority (3) confirmed (3) needs-reproduction (1)
Pull Request Labels

Packages

  • Total packages: 2
  • Total downloads: unknown
  • Total dependent packages: 0
    (may contain duplicates)
  • Total dependent repositories: 0
    (may contain duplicates)
  • Total versions: 16
proxy.golang.org: github.com/svenmarcus/hpc-rocket
  • Versions: 8
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 6.5%
Average: 6.7%
Dependent repos count: 6.9%
Last synced: 6 months ago
proxy.golang.org: github.com/SvenMarcus/hpc-rocket
  • Versions: 8
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 6.5%
Average: 6.7%
Dependent repos count: 6.9%
Last synced: 6 months ago

Dependencies

.github/workflows/build-docs.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
  • peaceiris/actions-gh-pages v3 composite
.github/workflows/python-app.yml actions
  • SonarSource/sonarcloud-github-action master composite
  • actions/checkout v2 composite
  • actions/download-artifact v2 composite
  • actions/setup-python v2 composite
  • actions/upload-artifact v2 composite
.github/workflows/codeql.yml actions
  • actions/checkout v3 composite
  • github/codeql-action/analyze v2 composite
  • github/codeql-action/autobuild v2 composite
  • github/codeql-action/init v2 composite
setup.py pypi
pyproject.toml pypi
  • PyYAML >=5.4.1
  • fs >=2.4.16
  • fs-sshfs >=1.0.1
  • paramiko >=2.10.4
  • rich >=10.1.0