https://github.com/bencardoen/launchjobs
Python script that runs arbitrary shell commands in parallel (for cluster/SLURM) usage
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (7.9%) to scientific vocabulary
Repository
Python script that runs arbitrary shell commands in parallel (for cluster/SLURM) usage
Basic Info
- Host: GitHub
- Owner: bencardoen
- License: gpl-3.0
- Language: Jupyter Notebook
- Default Branch: master
- Size: 20.5 KB
Statistics
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
README
Launching processes in parallel
This is a simple collection of bash/python scripts that allow you to run up to k processes in parallel, typically on a cluster. At the end of execution, the exitcodes of the processes are reported to you.
A typical use case is to run grid search, or process a whole set of files, and so on.
To avoid swamping the OS/RAM/IO only k processes are run in parallel.
Requirements
- Unix environment
- Python >= 3.5
Getting started
bash
$python deploy.py
This will run 15 jobs, executing a very simple bash script with a grid search of parameters.
Running your own code
Create a config.json (see demo.ipynb for simple way to do this)
- This should specify at least
- script path
- script name
- batch size : how many processes to execute in parallel
Change deploy.py for the parameters you want to pass, if any
On a slurm cluster
- Edit job.sh with your account credentials
bash $sbatch job.sh
Owner
- Name: Ben Cardoen
- Login: bencardoen
- Kind: user
- Location: Vancouver
- Company: https://github.com/sfu-mial
- Twitter: BenCardoen
- Repositories: 29
- Profile: https://github.com/bencardoen
PhD Student Computing Science @sfu-mial Simon Fraser University
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: 12 months ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0