starvz

R-based visualization techniques for the StarPU runtime

https://github.com/schnorr/starvz

Repository

R-based visualization techniques for the StarPU runtime

README.org

# -*- coding: utf-8 -*-"
#+STARTUP: overview indent

#+TITLE: StarVZ

#+OPTIONS: html-link-use-abs-url:nil html-postamble:auto
#+OPTIONS: html-preamble:t html-scripts:t html-style:t
#+OPTIONS: html5-fancy:nil tex:t
#+HTML_DOCTYPE: xhtml-strict
#+HTML_CONTAINER: div
#+DESCRIPTION:
#+KEYWORDS:
#+HTML_LINK_HOME:
#+HTML_LINK_UP:
#+HTML_MATHJAX:
#+HTML_HEAD:
#+HTML_HEAD_EXTRA:
#+SUBTITLE:
#+INFOJS_OPT:
#+CREATOR: Emacs 25.2.2 (Org mode 9.0.1)
#+LATEX_HEADER:
#+EXPORT_EXCLUDE_TAGS: noexport
#+EXPORT_SELECT_TAGS: export
#+TAGS: noexport(n) deprecated(d)

#+html: 
#+html:   
#+html: 
#+html: 
#+html:   
#+html: 
#+html: 
#+html:   
#+html: 
#+html: 
#+html:   
#+html: 
#+html: 
#+html:  
#+html: 
#+html: 



StarVZ consists in a performance analysis workflow that combines the
power of the R language (and the =tidyverse= realm) and many auxiliary
tools to provide a consistent, flexible, extensible, fast, and
versatile framework for the performance analysis of task-based
applications that run on top of the [[http://starpu.gforge.inria.fr/][StarPU runtime]] (with its MPI layer
for multi-node support). Its goal is to provide a fruitful
prototypical environment to conduct performance analysis
hypothesis-checking for task-based applications that run on
heterogeneous (multi-GPU, multi-core) multi-node HPC platforms.

The source code of this framework is released under the GPLv3 license.

** Instalation
#+begin_src R :results output :session *R* :exports both
# To install the R package
install.packages("devtools")
devtools::install_github("schnorr/starvz")
#+end_src

*** Non-R Dependencies for the whole workflow
- =starpu_fxt_tool=: Provided by StarPU
- =pj_dump=: Provided by PageNG
- =rec2csv=: Provided by recutils

** Quick Usage
*** Full Workflow - From FxT to Visualization
With =starpu_fxt_tool=, =pj_dump=, and =rec2csv= in =$PATH=:

#+begin_src bash :results output
export EXP_FOLDER=/folder_to_fxt_files/

export STARVZ_TOOLS=$(Rscript -e 'cat(system.file("tools/", package = "starvz"), sep="\n")')

$STARVZ_TOOLS/starvz $EXP_FOLDER
#+end_src

This generates =$EXP_FOLDER/starvz.png=

This is equivalent to:

#+begin_src shell :results output :exports both
$STARVZ_TOOLS/phase1-workflow.sh $EXP_FOLDER
$STARVZ_TOOLS/phase2-workflow.R $EXP_FOLDER $STARVZ_TOOLS/../etc/default.yaml
#+end_src

The tools will be available after instalation inside the package
folder subdirectory =tools/=.

This can be retrived by:

#+begin_src bash :results output
Rscript -e 'cat(system.file("tools/", package = "starvz"), sep="\n")'
#+end_src

*** Phase 2 (/Visualizations/) in R:
#+begin_src R :results output :session *R* :exports both
library(starvz)
data <- starvz_read("/folder_to_fxt_files/")
plot <- starvz_plot(data)
plot
#+end_src

=plot= will be a =ggplot= object that can be saved with =ggsave=

** Extended Content
- [[./INSTALL.org][Installation Instructions]]
- [[./PHASE1.org][Phase 1 Workflow Instructions]]
- Phase 2 Workflow Instructions

** Origin and Publications

A preliminary version of this framework has been released in the
companion website (check the reproducible paper link below) of the VPA
2016 workshop (held during the SC16 conference).  A second release of
the framework is available in the companion website of an accepted
article to Wiley's Concurrent and Computation: Practice and
Experience. A memory analysis extension was developed and discussed in
the third publication at CCGRID 2019.

- *Analyzing Dynamic Task-Based Applications on Hybrid Platforms: An
  Agile Scripting Approach*, Vinicius Garcia Pinto, Luka Stanisic,
  Arnaud Legrand, Lucas Mello Schnorr, Samuel Thibault, Vincent
  Danjean, in /2016 Third Workshop on Visual Performance Analysis
  (VPA@SC)/, Salt Lake City, UT, 2016, pp. 17-24.
  - [[https://doi.org/10.1109/VPA.2016.008][DOI]] and [[http://perf-ev-runtime.gforge.inria.fr/vpa2016/][Reproducible Paper]]

- *A Visual Performance Analysis Framework for Task-based Parallel
  Applications running on Hybrid Clusters*, Vinicius Garcia Pinto,
  Lucas Mello Schnorr, Luka Stanisic, Arnaud Legrand, Samuel Thibault,
  Vincent Danjean, in /Concurrency and Computation: Practice and
  Experience/, Wiley, 2018, 30 (18), pp.1-31.
  - [[https://dx.doi.org/10.1002/cpe.4472][DOI]], [[https://hal.inria.fr/hal-01616632/][Draft]], and [[https://gitlab.in2p3.fr/schnorr/ccpe2017][Companion website]]

- *Visual Performance Analysis of Memory Behavior in a Task-Based
  Runtime on Hybrid Platforms*, Lucas Leandro Nesi, Samuel Thibault,
  Luka Stanisic and Lucas Mello Schnorr, in /2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing
  (CCGRID)/, Larnaca, Cyprus, 2019, pp. 142-151.
  - [[https://dx.doi.org/10.1109/CCGRID.2019.00025][DOI]] and [[https://gitlab.com/lnesi/starpu_mem_analysis][Companion website]]

** Docker container

Please [[./Dockerfile][check this DockerFile]] to create a docker container with all the
necessary requirements for a basic utilization of the starvz framework
(in the form of an R package). Assuming that you have =docker= installed
in your system, you may want to simply pull and run this container
from Docker Hub, like this:

#+begin_src shell :results output
docker pull schnorr/starvz
docker run -it schnorr/starvz
#+end_src

After entering the container, run R and load the =starvz= package with:

#+begin_src R :results output :session :exports both
library(starvz)
#+end_src

** Team and Contact

- [[http://www.inf.ufrgs.br/~schnorr][Lucas Mello Schnorr (schnorr@inf.ufrgs.br)]]
- Vinicius Garcia Pinto (vinicius.pinto@inf.ufrgs.br)
- Lucas Leando Nesi (lucas.nesi@inf.ufrgs.br)
- Marcelo Cogo Miletto (mcmiletto@inf.ufrgs.br)

** Getting help

Fell free to post an [[https://github.com/schnorr/starvz/issues][issue here in GitHub]].

Owner

GitHub Events

Committers

Last synced: over 2 years ago

Issues and Pull Requests

Last synced: almost 2 years ago

Packages