Science Score: 54.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
3 of 5 committers (60.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (17.1%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: icui
- License: mit
- Language: Python
- Default Branch: main
- Size: 9.35 MB
Statistics
- Stars: 2
- Watchers: 1
- Forks: 3
- Open Issues: 8
- Releases: 0
Metadata Files
README.md
nnodes
Nnodes is a simple workflow manager for Python functions and command line tools. It makes your life easier when running complicated jobs either in your local computer or in a large-scale cluster.
Intro slides (PDF), poster (PDF)
Quick start
Install
sh
pip install nnodes
Run example workflow (Gaussian inversion, requires numpy)
sh
git clone https://github.com/icui/nnodes
cd nnodes/examples/gaussian
nnrun
Check out Get started for details.
Features
Progress control. No job progress will be lost in nnodes. The progress management is adaptable so that an interrupted workflow can be stopped and resumed at any point, and it is possible to rewind to a previous state if any parameter does not turn out to perform well; running a partial workflow or merging multiple workflows are also supported.
MPI execution. Parallel execution of MPI tasks is easy and no manual configuration is required. An MPI task from any part of the workflow will be sent to an MPI executor and be executed whenever the cluster resources is available. This makes sure that the node hours are fully utilized.
Parameter management. Inspired by HTML document, nnodes introduces a hierarchical parameter system that simplifies the process of passing parameters to functions. A parameter in the parent node will by default be propagated to the child node, unless overwritten. This eliminates the need to pass the same parameter to different functions under the same parent node.
Why nnodes?
Workflow manager is essential for many scientific applications and there is a large number of existing workflow managers available. Many of them are mature and well maintained (see Workflows Community for a curated list and Existing Workflow Systems for a comprehensive list). However, we believe that nnodes still has unique advantages. In short, it is simpler than most general-purpose workflow managers and more flexible than most problem-specific workflow managers.
- Simplicity. Most professional workflow managers have very steep learning curves, and are sometimes deeply bound with specific computing architectures. Nodes, on the other hand, provides a unified interface for all operations and utilizes only high level APIs. Migrating existing workflows to nnodes is seamless in most cases.
- Flexibility. Nnodes is not tied to a specific scientific problem, and it is decided by the user how deeply they wish to integrate their projects with nnodes. Users can simply use nnodes as a progress controller, or MPI executor, which requires little code change, or they can go so far as to let nnodes manage their entire project.
- Portability. Nnodes currently supports Slurm and LSF systems but also has API for users to define their custom environment. The workflow is saved in a single pickle file that can be transferred to a new system and continue from where it was left off. The MPI executor adapts automatically so no manual configuration is needed to utilize the full cluster resources.
Alternatives
If you are looking for more options, below are some projects worth checking out:
Contact
If you have any questions, please submit a GitHub issue or send an email.
Owner
- Name: Congyue Cui
- Login: icui
- Kind: user
- Location: Princeton, NJ, 08540
- Company: Princeton University
- Repositories: 37
- Profile: https://github.com/icui
Citation (CITATION.cff)
cff-version: 1.2.0 authors: - family-names: "Cui" given-names: "Congyue" - family-names: "Lucas" given-names: "Sawade" - family-names: "Jeroen" given-names: "Tromp" title: "Nnodes, a workflow manager for HPC clusters" version: 0.0.1 date-released: 2022-07-20 url: "https://github.com/icui/nnodes"
GitHub Events
Total
Last Year
Committers
Last synced: about 3 years ago
All Time
- Total Commits: 109
- Total Committers: 5
- Avg Commits per committer: 21.8
- Development Distribution Score (DDS): 0.404
Top Committers
| Name | Commits | |
|---|---|---|
| Congyue Cui | c****i@p****u | 65 |
| Lucas Sawade | l****e@p****u | 30 |
| Congyue Cui | 4****i@u****m | 12 |
| Wenjie Lei | l****i@p****u | 1 |
| Michael R. Crusoe | 1****c@u****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 7 months ago
All Time
- Total issues: 13
- Total pull requests: 34
- Average time to close issues: 4 months
- Average time to close pull requests: about 2 hours
- Total issue authors: 3
- Total pull request authors: 4
- Average comments per issue: 0.85
- Average comments per pull request: 0.09
- Merged pull requests: 30
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- icui (6)
- lsawade (5)
- wjlei1990 (2)
Pull Request Authors
- icui (19)
- lsawade (12)
- mr-c (1)
- wjlei1990 (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 12 last-month
- Total dependent packages: 0
- Total dependent repositories: 1
- Total versions: 2
- Total maintainers: 1
pypi.org: nnodes
A workflow manager for clusters.
- Homepage: https://github.com/icui/nnodes
- Documentation: https://nnodes.readthedocs.io/
- License: MIT License
-
Latest release: 0.0.2
published about 4 years ago