Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/darma-tasking/lb-analysis-framework

Analysis framework for exploring, testing, and comparing load balancing strategies

https://github.com/darma-tasking/lb-analysis-framework

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: ieee.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (16.6%) to scientific vocabulary

Keywords

concurrency distributed-computing hpc load-balancing parallelism
Last synced: 5 months ago · JSON representation

Repository

Analysis framework for exploring, testing, and comparing load balancing strategies

Basic Info
  • Host: GitHub
  • Owner: DARMA-tasking
  • License: other
  • Language: Python
  • Default Branch: develop
  • Homepage:
  • Size: 32.4 MB
Statistics
  • Stars: 4
  • Watchers: 1
  • Forks: 2
  • Open Issues: 19
  • Releases: 3
Topics
concurrency distributed-computing hpc load-balancing parallelism
Created about 7 years ago · Last pushed 7 months ago
Metadata Files
Readme License

README.md

Unit Tests Pylint Coverage

This is the repository for the Load-Balancing Analysis Framework (LBAF)

It contains the following subdirectories:

  • src: Load-Balancing Simulator code
  • doc: Research papers and related documents
  • data: Various data inputs or outputs
  • tests: Unit tests and acceptance tests

Please refer to our documentation for more details.

Table of Contents

Getting Started

LBAF currently supports Python 3.8 - 3.11. You can download Python here.

Optional: Create a virtual environment (recommended in development)

To create and activate a virtual environment: shell python -m venv venv source venv/bin/activate

[!NOTE] You can create separate virtual environments for different development branches. For example, a Python 3.8 environment for branch 125 could be named venv38-branch-125. Within this environment, you can install lbaf as an editable package (see below).

Installation

LBAF can be installed in two ways:

1. Install the LBAF Package (recommended)
Users can easily install the latest release of LBAF with: ```shell pip install lbaf ``` Developers should clone the repo and install the package in editable mode: ```shell git clone git@github.com:DARMA-tasking/LB-analysis-framework.git pip install -e LB-analysis-framework ```
2. Install Dependencies
If you do not wish to install LBAF as a package, simply clone the repo and install dependencies: ```shell git clone git@github.com:DARMA-tasking/LB-analysis-framework.git pip install -r LB-analysis-framework/requirements.txt ```

Testing

Begin by installing the test dependencies in requirements.txt.

shell pip install tox coverage pylint pytest anybadge

Then, to run all tests locally:

shell cd <project-path> tox -e py<x>

where <x> is 38, 39, 310, or 311, depending on your Python version. For example, in an environment with Python 3.8: tox -e py38.

The tox command will: - run all tests defined in tox.ini - create the artifacts directory in main project path - create an html coverage report and a pylint report within the artifacts directory

Usage

If the lbaf package is installed, LBAF can be run using the following command:

shell lbaf -c <config-file-path>

If dependencies were installed instead, LBAF must be run from source:

shell cd <project-path> python src/lbaf/Applications/LBAF_app.py -c <config-file-path>

Configuration File

The configuration file is a YAML file that specifies how LBAF will run.

<config-file-path> can be an absolute path or a relative path to your configuration file.

A description of each parameter in the configuration file can be found here, and sample configurations can be found in the config directory.

Visualization

LBAF can optionally leverage vt-tv, a DARMA-tasking tool built off of VTK, to visualize the work-to-rank mappings, communications, and memory usage of a run.

To get started, you will need to build VTK (instructions here).

Then, clone the vt-tv repository and install the Python bindings:

shell git clone https://github.com/DARMA-tasking/vt-tv.git VTK_DIR=/path/to/vtk/build pip install vt-tv

Once vt-tv has been installed, you may include visualization parameters in the configuration file. Sample parameters are found (commented out) at the bottom of config/conf.yaml.

For more instructions on building and using vt-tv, refer to the documentation.

Verbosity

To print a list of all Quantities of Interest (QOI) supported by LBAF, add a verbosity argument to the run command:

shell cd <project-path> lbaf -c <config-file-name> -v <verbosity-level>

or

shell cd <project-path> python src/lbaf/Applications/LBAF_app.py -c <config-file-name> -v <verbosity-level>

To output only the Rank QOI, use -v 1. Otherwise, to print both Rank and Object QOI, use -v 2.

Additional Programs

JSON data files validator
JSON data files Validator validates vt data files against defined schema. It is located in the vt repository and can be found [here](https://raw.githubusercontent.com/DARMA-tasking/vt/develop/scripts/JSON_data_files_validator.py). #### Download into LBAF If the `lbaf` package is installed, run: ```shell lbaf-vt-data-files-validator-loader ``` Otherwise, run from source: ```shell cd python src/lbaf/Utils/lbsJSONDataFilesValidatorLoader.py ``` The script will be saved to `/src/lbaf/imported/JSON_data_files_validator.py` #### Run from LBAF If the `lbaf` package is installed, run: ```shell lbaf-vt-data-files-validator ``` Otherwise, run from source: ```shell cd python src/lbaf/imported/JSON_data_files_validator.py ``` _Note: This command automatically downloads the `JSON_data_files_validator.py` script if needed._ #### Usage These commands assume that LBAF was installed as a package. When running from source, replace the run command as noted above. For single file: ```shell # With relative path lbaf-vt-data-files-validator --file_path=../../../data/nolb-8color-16nodes-data/data.0.json # With absolute path lbaf-vt-data-files-validator --file_path=/data/nolb-8color-16nodes-data/data.0.json ``` For many files in the same directory: ```shell # With relative path lbaf-vt-data-files-validator --dir_path=../../../data/nolb-8color-16nodes-data # With absolute path lbaf-vt-data-files-validator --dir_path=/data/nolb-8color-16nodes-data # Optionally one could pass --file_prefix and/or --file_suffix # When one passes files with given prefix/suffix or both will be validated # When no prefix and suffix will be given validator will find most common prefix and suffix in the directory # and will use them for validation process lbaf-vt-data-files-validator --dir_path=../../data/nolb-8color-16nodes-data --file_prefix=data --file_suffix=json ```
vt Data Extractor
The vt Data Extractor extracts phases from vt stats files. #### Usage To run using the lbaf package: ```shell lbaf-vt-data-extractor ``` To run from source: ```shell cd python src/lbaf/Utils/lbsVTDataExtractor.py ``` #### Input arguments * `input_data_dir`: str - path to dir with files to extract e.g. `"./data/"` * `output_data_dir`: str - path to dir where files should be saved e.g. `"./output"` (will be created when doesn't exist) * `phases_to_extract`: list - list of phases `[int or str]` e.g. `[0, 1, "2-4"]` will extract phases `[0, 1, 2, 3, 4]` * `file_prefix`: str - data file prefix e.g. if filename is `stats.0.json`, then prefix should be set to "stats" * `file_suffix`: str - data file suffix e.g. if filename is `stats.0.json`, then suffix should be set to "json" * `compressed`: bool - when True, brotli must be imported and then output data will be compressed * `schema_type`: str - should be `"LBDatafile"` or `"LBStatsfile"` depends on input data. Only `"LBStatsfile"` is supported * `check_schema`: bool - when True, validates schema (more time-consuming)
vt Data Maker
The vt Data Maker generates a dataset of JSON files that may be used throughout the DARMA-tasking organization. The generated files are compatible with `LBAF`, `vt-tv`, and `vt`. If the `lbaf` package is installed, run with: ```sh lbaf-vt-data-files-maker ``` Otherwise, run: ```shell python src/lbaf/Utils/lbsJSONDataFilesMaker.py ``` The program can be run interactively with the `--interactive` argument. Otherwise, it accepts a pre-written specification file (`--spec-file`) and the file stem for the resulting data files (`--data-stem`). Further documentation, including usage and examples, can be found within the script itself.

LBAF in Literature

Optimizing Distributed Load Balancing for Workloads with Time-Varying Imbalance

"This paper explores dynamic load balancing algorithms used by asynchronous many-task (AMT), or ‘task-based’, programming models to optimize task placement for scientific applications with dynamic workload imbalances."

Owner

  • Name: DARMA
  • Login: DARMA-tasking
  • Kind: organization
  • Location: Sandia National Laboratories

Asynchronous task-based programming and execution models

GitHub Events

Total
  • Create event: 25
  • Release event: 1
  • Issues event: 49
  • Watch event: 3
  • Delete event: 24
  • Issue comment event: 46
  • Push event: 180
  • Pull request review comment event: 35
  • Pull request review event: 64
  • Pull request event: 49
  • Fork event: 1
Last Year
  • Create event: 25
  • Release event: 1
  • Issues event: 49
  • Watch event: 3
  • Delete event: 24
  • Issue comment event: 46
  • Push event: 180
  • Pull request review comment event: 35
  • Pull request review event: 64
  • Pull request event: 49
  • Fork event: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 18
  • Total pull requests: 17
  • Average time to close issues: 2 months
  • Average time to close pull requests: 16 days
  • Total issue authors: 3
  • Total pull request authors: 4
  • Average comments per issue: 0.44
  • Average comments per pull request: 0.94
  • Merged pull requests: 12
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 18
  • Pull requests: 17
  • Average time to close issues: 2 months
  • Average time to close pull requests: 16 days
  • Issue authors: 3
  • Pull request authors: 4
  • Average comments per issue: 0.44
  • Average comments per pull request: 0.94
  • Merged pull requests: 12
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • ppebay (39)
  • cwschilly (15)
  • lifflander (8)
  • nmm0 (1)
  • pierrepebay (1)
Pull Request Authors
  • cwschilly (27)
  • ppebay (25)
  • lifflander (7)
  • pierrepebay (3)
  • tlamonthezie (2)
  • dependabot[bot] (1)
Top Labels
Issue Labels
enhancement (30) bug (11) low-priority (5) question (2) high-priority (1)
Pull Request Labels
bug (3) enhancement (2) dependencies (1) low-priority (1)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 10 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 2
  • Total maintainers: 1
pypi.org: lbaf

Analysis framework for exploring, testing, and comparing load balancing strategies

  • Versions: 2
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 10 Last month
Rankings
Dependent packages count: 7.3%
Forks count: 22.7%
Stargazers count: 27.9%
Average: 31.6%
Dependent repos count: 68.6%
Maintainers (1)
lifflander
Last synced: 6 months ago

Dependencies

docker-compose.yml docker
  • ${REPO} lbaf
pyproject.toml pypi
requirements.txt pypi
  • Brotli ==1.0.9
  • Jinja2 ==3.1.2
  • PyYAML ==6.0
  • Pygments ==2.15.0
  • anybadge ==1.9.0
  • build ==0.7.0
  • colorama ==0.4.6
  • coverage ==6.3.2
  • docutils ==0.19
  • matplotlib ==3.5.3
  • matplotlib ==3.6.2
  • numpy ==1.22.3
  • pylint ==2.12.2
  • pytest ==7.1.1
  • schema ==0.7.5
  • scikit-learn ==1.0.2
  • tox ==4.6.0
  • vtk ==9.1.0
  • vtk ==9.0.1
.github/workflows/action-git-diff-check.yml actions
  • actions/checkout v2 composite
  • joel-coffman/action-git-diff-check 0.1 composite
.github/workflows/build-and-publish-to-pypi.yml actions
  • actions/checkout master composite
  • actions/setup-python v4 composite
  • pypa/gh-action-pypi-publish release/v1 composite
.github/workflows/check-commit-format.yml actions
  • DARMA-tasking/check-commit-format master composite
  • actions/checkout v2 composite
.github/workflows/check-pr-fixes-issue.yml actions
  • DARMA-tasking/check-pr-fixes-issue master composite
.github/workflows/deploy_docs.yml actions
  • JamesIves/github-pages-deploy-action v4 composite
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/find-trailing-whitespace.yml actions
  • DARMA-tasking/find-trailing-whitespace master composite
  • actions/checkout v2 composite
.github/workflows/find-unsigned-commits.yml actions
  • DARMA-tasking/find-unsigned-commits master composite
.github/workflows/code-quality.yml actions
  • JamesIves/github-pages-deploy-action v4 composite
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
  • actions/upload-artifact v3 composite
  • phoenix-actions/test-reporting v12 composite