dcsim - DCSim simulator release v0.3

Production ready version of a distributed computing simulator (DCSim) based on WRENCHv2.2 and SimGridv3.34. Allows to simulate freely selectable workflows of batch and streaming jobs on arbitrary computing platforms processing freely selectable (amounts of) data. Thanks to performance updates in WRENCH and SimGrid the memory and computational requirements on executing resources for the simulator could be improved. This version is used in the studies presented in https://indico.jlab.org/event/459/contributions/11490/.

What's Changed

• Extension the support for platforms consisting of arbitrary numbers of GRID storages and workers by @HerrHorizontal in https://github.com/HEPCompSim/DCSim/pull/24
• Enable fluid model in file read by @HerrHorizontal in https://github.com/HEPCompSim/DCSim/pull/47
• Additional features for the workload generation by @ArturAkh in https://github.com/HEPCompSim/DCSim/pull/49 by @HerrHorizontal in https://github.com/HEPCompSim/DCSim/pull/51
• Extend the sampling of job characteristics by @HerrHorizontal in https://github.com/HEPCompSim/DCSim/pull/54
• Config and plot changes for exemplary scenarios by @HerrHorizontal in https://github.com/HEPCompSim/DCSim/pull/56
• Enable simulation of scheduling and executing multi-core jobs by @HerrHorizontal in https://github.com/HEPCompSim/DCSim/pull/57
• Enable dataset configuration by @rsreds in https://github.com/HEPCompSim/DCSim/pull/53
• Rework simulator configuration to be completely steering-file based
• Update plotting and helper scripts
• Extended examples
• Minor bug fixes

New Contributors

• @rsreds made their first contribution in https://github.com/HEPCompSim/DCSim/pull/52

Full Changelog: https://github.com/HEPCompSim/DCSim/compare/v0.2...v0.3

- C++
Published by HerrHorizontal over 2 years ago

dcsim - SG-Batch-simulator pre-release v0.2

Completed implementation of the SG-machines-batch benchmark simulator based on WRENCH-2.0. It uses a new job concept of active jobs available in WRENCH-2.0 to enable the simulation of workflows consisting of jobs, which stream their input files, while simultaneously processing the information or jobs, which copy their input-files previous to computation.

The plotting and analysis tools have been updated to support the new job concept.

Thanks to performance improvements in WRENCH-2.0 the scaling of the simulation has also improved significantly (see assets below), when compared to previous releases (approx. factor 10 for memory, factor 0.8 for simulation time).

What's Changed

• New job concept of active jobs performing actions by @HerrHorizontal and @henricasanova in https://github.com/HEPCompSim/DistCacheSim/pull/15
• Support of jobs streaming input files by @HerrHorizontal
• Improvements and adjustments in plotting and analysis tools by @HerrHorizontal and @ArturAkh

Full Changelog: https://github.com/HEPCompSim/DistCacheSim/compare/v0.1-hacky...v0.2

- C++
Published by HerrHorizontal about 4 years ago

dcsim - DistCachSim SGBatch-simulator pre-release v0.1

First edition of the SG-machines-batch benchmark simulator. This is based on a private hacky version of WRENCH-1.10. Due to a probable bug in the workflow builder (see issue #7) only the simulation of copy-jobs gives meaningful results. For the simulation of streaming-jobs, newer versions of this simulator based on WRENCH-2.0 must be considered.

However, with this release the basic foundations for the simulation of copy-jobs and the analysis of the simulation outputs are given (see attached figures).

Also, it is possible to perform a simple study on the runtime and memory consumption of the simulation with increasing numbers of jobs (see attached figure).

What's Changed

• XRootD-ish blockwise data streaming by @HerrHorizontal in https://github.com/HerrHorizontal/DistCacheSim/pull/2. However this is bugged and should not be used, see issue #7
• Test/sgbatch henri small fix by @henricasanova in https://github.com/HerrHorizontal/DistCacheSim/pull/3
• Refactor simulation monitoring dump/output and memory usage improvements by @HerrHorizontal in https://github.com/HerrHorizontal/DistCacheSim/pull/4
• CLI with the help of boost::program_options by @HerrHorizontal in https://github.com/HerrHorizontal/DistCacheSim/pull/9
• Plotting tools by @HerrHorizontal in https://github.com/HerrHorizontal/DistCacheSim/pull/13
• Checkout script for easy installation by @HerrHorizontal in https://github.com/HerrHorizontal/DistCacheSim/pull/14
• SG-machines-batch benchmark simulator by @HerrHorizontal in https://github.com/HerrHorizontal/DistCacheSim/pull/1

New Contributors

• @HerrHorizontal made their first contribution in https://github.com/HerrHorizontal/DistCacheSim/pull/2
• @henricasanova made their first contribution in https://github.com/HerrHorizontal/DistCacheSim/pull/3
• @ArturAkh

Full Changelog: https://github.com/HerrHorizontal/DistCacheSim/commits/v0.1

- C++
Published by HerrHorizontal about 4 years ago