hatchet

Graph-indexed Pandas DataFrames for analyzing hierarchical performance data

https://github.com/llnl/hatchet

Science Score: 85.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • CITATION.cff file
    Found CITATION.cff file
  • codemeta.json file
    Found codemeta.json file
  • .zenodo.json file
    Found .zenodo.json file
  • DOI references
    Found 1 DOI reference(s) in README
  • Academic publication links
    Links to: acm.org
  • Committers with academic emails
    10 of 22 committers (45.5%) from academic institutions
  • Institutional organization owner
    Organization llnl has institutional domain (software.llnl.gov)
  • JOSS paper metadata
  • Scientific vocabulary similarity
    Low similarity (21.7%) to scientific vocabulary

Keywords

comparative-analysis data-analytics graphs hierarchical-data hpc performance performance-analysis python radiuss trees
Last synced: 6 months ago · JSON representation ·

Repository

Graph-indexed Pandas DataFrames for analyzing hierarchical performance data

Basic Info
Statistics
  • Stars: 33
  • Watchers: 6
  • Forks: 19
  • Open Issues: 35
  • Releases: 14
Topics
comparative-analysis data-analytics graphs hierarchical-data hpc performance performance-analysis python radiuss trees
Created about 4 years ago · Last pushed 8 months ago
Metadata Files
Readme Changelog License Citation

README.md

hatchet Hatchet

Build Status Read the Docs codecov Code Style: Black

Hatchet is a Python-based library that allows Pandas dataframes to be indexed by structured tree and graph data. It is intended for analyzing performance data that has a hierarchy (for example, serial or parallel profiles that represent calling context trees, call graphs, nested regions’ timers, etc.). Hatchet implements various operations to analyze a single hierarchical data set or compare multiple data sets, and its API facilitates analyzing such data programmatically.

To use hatchet, install it with pip:

$ pip install llnl-hatchet

Or, if you want to develop with this repo directly, run the install script from the root directory, which will build the cython modules and add the cloned directory to your PYTHONPATH:

$ source install.sh

Documentation

See the Getting Started page for basic examples and usage. Full documentation is available in the User Guide.

Examples of performance analysis using hatchet are available here.

The hatchet tutorial provides a step by step introduction to hatchet features through a live interactive environment in BinderHub.

Interactive Visualization

Hatchet now provides an interactive visualization which can be run inside of your Jupyter notebooks. It is dependant on different mechanisms for building which we describe here:

The software in the hatchet/vis subdirectory requires Node.js and Node Package Manager for the development and building of javascript code.

Installing node packages

One you have Node and NPM installed on your system, you can install all necessary node packages by running the following line in your terminal from the hatchet/vis/ directory:

npm install

Building out javascript code with webpack

To build out javascript into the static bundles used by the Jupyter visualizations, run the following line from the hatchet/vis directory in your terminal:

npm run build

Alterntinatively if you are developing a visualization, you can run the following line to force bundles to automatically update when you change the javascript source code:

npm run watch

Contributing

Hatchet is an open source project. We welcome contributions via pull requests, and questions, feature requests, or bug reports via issues.

Authors

Many thanks go to Hatchet's contributors.

Citing Hatchet

If you are referencing Hatchet in a publication, please cite the following paper:

  • Abhinav Bhatele, Stephanie Brink, and Todd Gamblin. Hatchet: Pruning the Overgrowth in Parallel Profiles. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '19). ACM, New York, NY, USA. DOI

License

Hatchet is distributed under the terms of the MIT license.

All contributions must be made under the MIT license. Copyrights in the Hatchet project are retained by contributors. No copyright assignment is required to contribute to Hatchet.

See LICENSE and NOTICE for details.

SPDX-License-Identifier: MIT

LLNL-CODE-741008

Owner

  • Name: Lawrence Livermore National Laboratory
  • Login: LLNL
  • Kind: organization
  • Email: github-admin@llnl.gov
  • Location: Livermore, CA, USA

For over 70 years, the Lawrence Livermore National Laboratory has applied science and technology to make the world a safer place.

Citation (CITATION.cff)

cff-version: 1.2.0
title: Hatchet
message: "If you use Hatchet, please cite it as below."
repository-code: https://github.com/LLNL/hatchet
preferred-citation:
  type: conference-paper
  doi: 10.1145/3295500.3356219
  url: https://github.com/LLNL/hatchet
  authors:
  - family-names: Bhatele
    given-names: Abhinav
  - family-names: Brink
    given-names: Stephanie
  - family-names: Gamblin
    given-names: Todd
  title: "Hatchet: Pruning the Overgrowth in Parallel Profiles"
  conference:
    name: "International Conference for High Performance Computing, Networking, Storage and Analysis (SC '19)"
    city: "Denver"
    region: "Colorado"
    country: "USA"
    date-start: 2019-11-17
    date-end: 2019-11-22
  year: 2019
  notes: LLNL-CODE-741008
  publisher:
    name: ACM
    city: "New York"
    region: "New York"
    country: "USA"

GitHub Events

Total
  • Create event: 10
  • Issues event: 4
  • Release event: 3
  • Watch event: 5
  • Delete event: 4
  • Issue comment event: 21
  • Push event: 14
  • Pull request review event: 17
  • Pull request review comment event: 7
  • Pull request event: 28
  • Fork event: 1
Last Year
  • Create event: 10
  • Issues event: 4
  • Release event: 3
  • Watch event: 5
  • Delete event: 4
  • Issue comment event: 21
  • Push event: 14
  • Pull request review event: 17
  • Pull request review comment event: 7
  • Pull request event: 28
  • Fork event: 1

Committers

Last synced: 11 months ago

All Time
  • Total Commits: 548
  • Total Committers: 22
  • Avg Commits per committer: 24.909
  • Development Distribution Score (DDS): 0.609
Past Year
  • Commits: 17
  • Committers: 4
  • Avg Commits per committer: 4.25
  • Development Distribution Score (DDS): 0.412
Top Committers
Name Email Commits
Abhinav Bhatele b****e@l****v 214
Stephanie Brink b****2@l****v 166
Ian Lumsden l****n@g****m 52
Todd Gamblin t****n@l****v 44
cscully-allison c****n@n****u 28
Michael McKinsey M****1@g****m 10
Dewi 4****n 7
Jonathan R. Madsen j****n 5
Onur Cankur o****r@u****u 4
Julius Plehn J****n 3
suraj jarus j****1@g****m 2
matthewkotila m****a@g****m 2
David Boehme b****3@l****v 2
Stephanie Brink l****e@g****m 1
Johannes Blaschke j****e@g****m 1
Kate Isaacs k****s@c****u 1
Katy Williams k****s@e****u 1
Omer-Sharif 7****f 1
dhruvnm d****7@g****m 1
draganagrbicllnl g****1@l****v 1
jonesholger j****r@g****m 1
pearce8 p****8@l****v 1
Committer Domains (Top 20 + Academic)

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 33
  • Total pull requests: 141
  • Average time to close issues: about 2 months
  • Average time to close pull requests: 30 days
  • Total issue authors: 11
  • Total pull request authors: 16
  • Average comments per issue: 0.61
  • Average comments per pull request: 0.61
  • Merged pull requests: 104
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 3
  • Pull requests: 26
  • Average time to close issues: about 17 hours
  • Average time to close pull requests: 30 days
  • Issue authors: 3
  • Pull request authors: 3
  • Average comments per issue: 0.33
  • Average comments per pull request: 0.69
  • Merged pull requests: 16
  • Bot issues: 0
  • Bot pull requests: 0
Top Authors
Issue Authors
  • slabasan (13)
  • ilumsden (9)
  • cscully-allison (3)
  • dejangrubisic (1)
  • michaelmckinsey1 (1)
  • chapman39 (1)
  • Jokeren (1)
  • balos1 (1)
  • PhilipDeegan (1)
  • jithunnair-amd (1)
  • mkre (1)
Pull Request Authors
  • ilumsden (54)
  • slabasan (50)
  • michaelmckinsey1 (34)
  • cscully-allison (8)
  • dyokelson (6)
  • Julius-Plehn (4)
  • jarusified (3)
  • draganaurosgrbic (2)
  • MichaelMcKinsey1 (2)
  • r-yin (2)
  • draganagrbicllnl (2)
  • spencer-gre (2)
  • Jokeren (1)
  • jonesholger (1)
  • pearce8 (1)
Top Labels
Issue Labels
priority-normal (5) type-feature (4) type-question (3) area-graphframe (3) area-deployment (2) area-docs (2) area-query-lang (2) type-bug (2) area-visualization (1) priority-urgent (1) type-internal-cleanup (1) area-readers (1)
Pull Request Labels
priority-normal (87) status-ready-for-review (58) type-bug (48) type-feature (36) area-ci (20) area-readers (20) area-deployment (20) area-graphframe (19) status-work-in-progress (18) status-approved (16) type-internal-cleanup (16) area-visualization (15) type-release (14) area-query-lang (14) priority-urgent (13) status-ready-to-merge (11) priority-high (9) area-docs (7) area-external (5) area-writers (5) area-utils (5) area-cython (5) type-style (4) status-help-wanted (3) area-tests (3) status-revisions-needed (2) status-wont-fix (2) status-blocked (1) documentation (1) enhancement (1)

Dependencies

.github/workflows/pip-install-tester.yaml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
.github/workflows/unit-tests.yaml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
  • codecov/codecov-action v3 composite
hatchet/external/roundtrip/package.json npm
  • @babel/core ^7.16.0 development
  • @babel/preset-env ^7.16.0 development
  • @babel/preset-react ^7.16.0 development
  • babel-loader ^8.2.3 development
  • html-webpack-plugin ^5.3.2 development
  • webpack ^5.58.2 development
  • webpack-cli ^4.9.0 development
  • css-loader ^6.5.0
  • d3 ^7.0.4
  • react ^17.0.2
  • react-dom ^17.0.2
  • style-loader ^3.3.1
hatchet/vis/package.json npm
  • @babel/core ^7.16.0 development
  • @babel/preset-env ^7.16.0 development
  • @babel/preset-react ^7.16.0 development
  • babel-loader ^8.2.3 development
  • eslint ^7.32.0 development
  • eslint-plugin-import ^2.25.3 development
  • eslint-plugin-node ^11.1.0 development
  • eslint-plugin-promise ^5.2.0 development
  • html-webpack-plugin ^5.3.2 development
  • webpack ^5.58.2 development
  • webpack-cli ^4.9.0 development
  • css-loader ^6.5.0
  • d3 ^7.0.4
  • d3-flextree ^2.1.2
  • d3-quadtree ^3.0.1
  • d3v4 ^4.2.2
  • style-loader ^3.3.1
docs/requirements.txt pypi
  • PyYAML *
  • caliper-reader *
  • cython *
  • matplotlib *
  • multiprocess *
  • numpy *
  • pandas *
  • pydot *
  • textX *
hatchet/external/roundtrip/requirements.txt pypi
  • beautifulsoup4 *
  • ipython *
  • numpy *
  • pandas *
  • pathlib *
hatchet/external/roundtrip/setup.py pypi
  • numpy *
requirements.txt pypi
  • PyYAML *
  • beautifulsoup4 *
  • caliper-reader *
  • codecov *
  • cython *
  • flake8 *
  • matplotlib *
  • multiprocess *
  • numpy *
  • pandas *
  • pep8-naming *
  • pydot *
  • pytest *
  • textX *
setup.py pypi
  • PyYAML *
  • caliper-reader *
  • matplotlib *
  • multiprocess *
  • numpy *
  • pandas *
  • pydot *
  • textX *
.github/workflows/build_and_upload_wheels.yaml actions
  • actions/checkout v3 composite
  • actions/download-artifact v3 composite
  • actions/upload-artifact v3 composite
  • docker/setup-qemu-action v2 composite
  • pypa/cibuildwheel v2.12.1 composite
  • pypa/gh-action-pypi-publish v1.5.0 composite
hatchet/external/roundtrip/pyproject.toml pypi
hatchet/vis/requirements.txt pypi
pyproject.toml pypi