https://github.com/carv-ics-forth/slurmdatamover
A tool for managing data dependencies between jobs on a Workflow.
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.6%) to scientific vocabulary
Repository
A tool for managing data dependencies between jobs on a Workflow.
Basic Info
- Host: GitHub
- Owner: CARV-ICS-FORTH
- License: bsd-3-clause
- Language: C++
- Default Branch: main
- Size: 7.06 MB
Statistics
- Stars: 1
- Watchers: 5
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Slurm Data Mover (SDM)
A tool for managing data dependencies between jobs on a Workflow. This tool is responsible sending data on nodes that are allocated for executing a job via Slurm.
SDM aims to enable users to express data dependencies between phases of their workflows by identifying input and output files at the level of jobs. SDM offers users directives to implement data staging, i.e. orchestrate the necessary data transfers so that the jobs that are part of a workflow will find their required datasets in-place once they are scheduled for execution. In this manner, Slurm becomes more aware of workflows and is able to effectively utilize node-local storage resources. The expected benefits are derived from reduction of storage I/O load, and potential contention, to the underlying shared parallel filesystems, leading to improved job and workflow runtimes for jobs and workflows, while reducing potential performance interference.
Requirements
SDM requires the following packages:
- cmake
- make
- gcc + g++
Optionaly:
- ccmake (configuration ui)
- libpoco
- doxygen (documentation)
ArchLinux
sudo pacman -S cmake # poco doxygen
CentOS
sudo yum install cmake # poco-devel poco-foundation poco-net doxygen
Ubuntu
sudo apt-get install cmake # libpoco-dev doxygen
Usage
``` Produce file.txt: sdm --produce file.txt Consume file.txt: sdm --consume file.txt Start Node Server: sdm --serve
Options:
-rhost:port --redis=host:port Redis socket address
-b
You must pass at least one of Consume,Produce,Serve or Wui. ```
Owner
- Name: Computer Architecture and VLSI Systems (CARV) Laboratory
- Login: CARV-ICS-FORTH
- Kind: organization
- Location: Heraklion, Greece
- Website: http://www.ics.forth.gr/carv/
- Repositories: 53
- Profile: https://github.com/CARV-ICS-FORTH
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0