Data

Tools

Indexes

Applications

Experiments

Open Source Science

qspool

Dependency-free solution to spool jobs into SLURM scheduler without exceeding queue capacity limits

https://github.com/mmore500/qspool

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 5 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.1%) to scientific vocabulary
Last synced: 6 months ago · JSON representation ·

Repository

Dependency-free solution to spool jobs into SLURM scheduler without exceeding queue capacity limits

Basic Info
  • Host: GitHub
  • Owner: mmore500
  • License: mit
  • Language: Python
  • Default Branch: master
  • Homepage:
  • Size: 88.9 KB
Statistics
  • Stars: 4
  • Watchers: 1
  • Forks: 0
  • Open Issues: 0
  • Releases: 1
Created about 3 years ago · Last pushed almost 2 years ago
Metadata Files
Readme License Citation

README.md

DOI PyPI Status CI Status

qspool is a dependency-free solution to spool jobs into SLURM scheduler without exceeding queue capacity limits.

Usage

You need to submit more slurm scripts than fit on the queue at once. bash tree . . ├── slurmscript0.slurm.sh ├── slurmscript1.slurm.sh ├── slurmscript2.slurm.sh ├── slurmscript3.slurm.sh ├── slurmscript4.slurm.sh ├── slurmscript5.slurm.sh ├── slurmscript6.slurm.sh ├── slurmscript7.slurm.sh ├── slurmscript8.slurm.sh ...

The qspool script will feed your job scripts onto the queue as space becomes available. bash python3 -m qspool *.slurm.sh

You can also provide job names via stdin, which is useful for very large job batches. bash find . -maxdepth 1 -name '*.slurm.sh' | python3 -m qspool

The qspool script creates a slurm job that submits your job scripts. When queue capacity fills, this qspool job will schedule a follow-up job to submit any remaining job scripts. This process continues until all job scripts have been submitted.

``` usage: qspool.py [-h] [--payload-job-script-paths-infile PAYLOADJOBSCRIPTPATHSINFILE] [--job-log-path JOBLOGPATH] [--job-script-cc-path JOBSCRIPTCCPATH] [--queue-capacity QUEUECAPACITY] [--qspooler-job-title QSPOOLERJOB_TITLE] [payloadjobscriptpaths ...]

positional arguments: payloadjobscript_paths What scripts to spool onto slurm queue? (default: None)

options: -h, --help show this help message and exit --payload-job-script-paths-infile PAYLOADJOBSCRIPTPATHSINFILE Where to read script paths to spool onto slurm queue? (default: <io.TextIOWrapper name='' mode='r' encoding='utf-8'>) --job-log-path JOBLOGPATH Where should logs for qspool jobs be written? (default: ~/joblog/) --job-script-cc-path JOBSCRIPTCCPATH Where should copies of submitted job scripts be kept? (default: ~/jobscript/) --queue-capacity QUEUECAPACITY How many jobs can be running or waiting at once? (default: 1000) --qspooler-job-title QSPOOLERJOB_TITLE What title should be included in qspooler job names? (default: none) ```

Installation

no installation: bash python3 "$(tmpfile="$(mktemp)"; curl -s https://raw.githubusercontent.com/mmore500/qspool/v0.5.0/qspool.py > "${tmpfile}"; echo "${tmpfile}")" [ARGS]

pip installation: bash python3 -m pip install qspool python3 -m qspool [ARGS]

qspool has zero dependencies, so no setup or maintenance is required to use it. Compatible all the way back to Python 3.6, so it will work on your cluster's ancient Python install.

How it Works

qspool * read contents of target slurm scripts * instantiate qspooler job script w/ target slurm scripts embedded * submit qspooler job script to slurm queue

⬇️ ⬇️ ⬇️

qspooler job 1 * submit embedded target slurm scripts one by one until queue is almost full * instantiate qspooler job script w/ remaining target slurm scripts embedded * submit qspooler job script to slurm queue

⬇️ ⬇️ ⬇️

qspooler job 2 * submit embedded target slurm scripts one by one until queue is almost full * instantiate qspooler job script w/ remaining target slurm scripts embedded * submit qspooler job script to slurm queue

...

qspooler job n * submit embedded target slurm scripts one by one * no embedded target slurm scripts remain * exit

Related Software

roll_q uses a similar approach to solve this problem. roll_q differs in implementation strategy. roll_q tracks submission progress via an index variable in a file associated with a job batch. qspool embeds jobs in the submission worker script itself.

Citing

If qspool is used in scientific publication, please cite it as

Matthew Andres Moreno (2024). mmore500/qspool. Zenodo. https://doi.org/10.5281/zenodo.10864602

bibtex @software{moreno2024qspool, author = {Matthew Andres Moreno}, title = {mmore500/qspool}, month = mar, year = 2024, publisher = {Zenodo}, doi = {10.5281/zenodo.10864602}, url = {https://zenodo.org/doi/10.5281/zenodo.10864602} }

And don't forget to leave a star on GitHub!

Owner

  • Name: Matthew Andres Moreno
  • Login: mmore500
  • Kind: user
  • Location: East Lansing, MI
  • Company: @devosoft

doctoral student, Computer Science and Engineering at Michigan State University

Citation (CITATION.cff) 

cff-version: 1.1.0
message: "If you use this software, please cite it as below."
title: 'qspool: a Python library to orchestrate SLURM jobs'
abstract: "qspool is a dependency-free solution to spool jobs into SLURM scheduler without exceeding queue capacity limits."
authors:
- family-names: Moreno
  given-names: Matthew Andres
  orcid: 0000-0003-4726-4479
date-released: 2023-03-23
doi: 10.5281/zenodo.10864602
license: MIT
repository-code: https://github.com/mmore500/qspool
url: "https://github.com/mmore500/qspool"
version: v1.0.1

GitHub Events

Total
Last Year

Committers

Last synced: 10 months ago

All Time
  • Total Commits: 74
  • Total Committers: 1
  • Avg Commits per committer: 74.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 1
  • Committers: 1
  • Avg Commits per committer: 1.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Matthew Andres Moreno m****g@g****m 74

Issues and Pull Requests

Last synced: 7 months ago

All Time
  • Total issues: 0
  • Total pull requests: 1
  • Average time to close issues: N/A
  • Average time to close pull requests: less than a minute
  • Total issue authors: 0
  • Total pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 1
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
  • mmore500 (1)
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 62 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 15
  • Total maintainers: 1
pypi.org: qspool

Dependency-free script to spool jobs into SLURM scheduler without exceeding queue capacity limits.

  • Versions: 15
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 62 Last month
Rankings
Dependent packages count: 10.1%
Dependent repos count: 21.6%
Average: 23.5%
Downloads: 38.9%
Maintainers (1)
mmore500
Last synced: 7 months ago