duct

Also see our Datalad Blog Post for a hands on example.

Installation

pip install con-duct

Quickstart

Try it out!

duct --sample-interval 0.5 --report-interval 1 test/data/test_script.py --duration 3 --memory-size=1000

duct is most useful when the report-interval is less than the duration of the script.

Summary:

A process wrapper script that monitors the execution of a command.

```shell

duct --help

usage: duct [-h] [--version] [-p OUTPUTPREFIX] [--summary-format SUMMARY_FORMAT] [--colors] [--clobber] [-l {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}] [-q] [--sample-interval SAMPLE_INTERVAL] [--report-interval REPORT_INTERVAL] [--fail-time FAIL_TIME] [-c {all,none,stdout,stderr}] [-o {all,none,stdout,stderr}] [-t {all,system-summary,processes-samples}] [-m MESSAGE] [--mode {new-session,current-session}] command [commandargs ...] ...

duct is a lightweight wrapper that collects execution data for an arbitrary command. Execution data includes execution time, system information, and resource usage statistics of the command and all its child processes. It is intended to simplify the problem of recording the resources necessary to execute a command, particularly in an HPC environment.

Resource usage is determined by polling (at a sample-interval). During execution, duct produces a JSON lines (see https://jsonlines.org) file with one data point recorded for each report (at a report-interval).

limitations: Duct uses session id to track the command process and its children, so it cannot handle the situation where a process creates a new session. If a command spawns child processes, duct will collect data on them, but duct exits as soon as the primary process exits.

environment variables: Many duct options can be configured by environment variables (which are overridden by command line options).

DUCTLOGLEVEL: see --log-level DUCTOUTPUTPREFIX: see --output-prefix DUCTSUMMARYFORMAT: see --summary-format DUCTSAMPLEINTERVAL: see --sample-interval DUCTREPORTINTERVAL: see --report-interval DUCTCAPTUREOUTPUTS: see --capture-outputs DUCT_MESSAGE: see --message

positional arguments: command [commandargs ...] The command to execute, along with its arguments. commandargs Arguments for the command.

options: -h, --help show this help message and exit --version show program's version number and exit -p OUTPUTPREFIX, --output-prefix OUTPUTPREFIX File string format to be used as a prefix for the files -- the captured stdout and stderr and the resource usage logs. The understood variables are {datetime}, {datetimefilesafe}, and {pid}. Leading directories will be created if they do not exist. You can also provide value via DUCTOUTPUTPREFIX env variable. (default: .duct/logs/{datetimefilesafe}-{pid}) --summary-format SUMMARYFORMAT Output template to use when printing the summary following execution. Accepts custom conversion flags: !S: Converts filesizes to human readable units, green if measured, red if None. !E: Colors exit code, green if falsey, red if truthy, and red if None. !X: Colors green if truthy, red if falsey. !N: Colors green if not None, red if None (default: Summary: Exit Code: {exitcode!E} Command: {command} Log files location: {logsprefix} Wall Clock Time: {wallclocktime:.3f} sec Memory Peak Usage (RSS): {peakrss!S} Memory Average Usage (RSS): {averagerss!S} Virtual Memory Peak Usage (VSZ): {peakvsz!S} Virtual Memory Average Usage (VSZ): {averagevsz!S} Memory Peak Percentage: {peakpmem:.2f!N}% Memory Average Percentage: {averagepmem:.2f!N}% CPU Peak Usage: {peakpcpu:.2f!N}% Average CPU Usage: {averagepcpu:.2f!N}% ) --colors Use colors in duct output. (default: False) --clobber Replace log files if they already exist. (default: False) -l {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}, --log-level {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG} Level of log output to stderr, use NONE to entirely disable. (default: INFO) -q, --quiet [deprecated, use log level NONE] Disable duct logging output (to stderr) (default: False) --sample-interval SAMPLEINTERVAL, --s-i SAMPLEINTERVAL Interval in seconds between status checks of the running process. Sample interval must be less than or equal to report interval, and it achieves the best results when sample is significantly less than the runtime of the process. (default: 1.0) --report-interval REPORTINTERVAL, --r-i REPORTINTERVAL Interval in seconds at which to report aggregated data. (default: 60.0) --fail-time FAILTIME, --f-t FAILTIME If command fails in less than this specified time (seconds), duct would remove logs. Set to 0 if you would like to keep logs for a failing command regardless of its run time. Set to negative (e.g. -1) if you would like to not keep logs for any failing command. (default: 3.0) -c {all,none,stdout,stderr}, --capture-outputs {all,none,stdout,stderr} Record stdout, stderr, all, or none to log files. You can also provide value via DUCTCAPTUREOUTPUTS env variable. (default: all) -o {all,none,stdout,stderr}, --outputs {all,none,stdout,stderr} Print stdout, stderr, all, or none to stdout/stderr respectively. (default: all) -t {all,system-summary,processes-samples}, --record-types {all,system-summary,processes-samples} Record system-summary, processes-samples, or all (default: all) -m MESSAGE, --message MESSAGE Record a descriptive message about the purpose of this execution. You can also provide value via DUCT_MESSAGE env variable. (default: ) --mode {new-session,current-session} Session mode: 'new-session' creates a new session for the command (default), 'current-session' tracks the current session instead of starting a new one. Useful for tracking slurm jobs or other commands that should run in the current session. (default: new-session)

``` <!-- END HELP -->

con-duct suite

In addition to duct, this project also includes a set of optional helpers under the con-duct command. These helpers may use 3rd party python libraries.

Installation

pip install con-duct[all]

Extras Helptext

```shell

con-duct --help

usage: con-duct [options]

A suite of commands to manage or manipulate con-duct logs.

positional arguments: {pp,plot,ls} Available subcommands pp Pretty print a JSON log. plot Plot resource usage for an execution. ls Print execution information for all matching runs.

options: -h, --help show this help message and exit -l {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG}, --log-level {NONE,CRITICAL,ERROR,WARNING,INFO,DEBUG} Level of log output to stderr, use NONE to entirely disable. --version show program's version number and exit

``` <!-- END EXTRAS HELP -->

FAQs

git-annex add keeps adding duct logs directly into git

By default, git-annex treats all dotfiles, and files under directories starting with a . as "small" regardless of annex.largefiles setting [ref: an issue describing the logic]. It is necessary to set annex.dotfiles variable to true to make git-annex treat them as regular files and thus subject to annex.largefiles setting [ref: git-annex config]. Could be done the repository (not just specific clone, but any instance since records in git-annex branch) wide using git annex config --set annex.dotfiles true.