@stdlib/bench-harness

Benchmark harness.

https://github.com/stdlib-js/bench-harness

Science Score: 54.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
✓
Academic publication links
Links to: arxiv.org
○
Committers with academic emails
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (15.2%) to scientific vocabulary

Keywords

bench benchmark harness javascript lib library measure node node-js nodejs perf performance standard stdlib tap

Keywords from Contributors

accumulator iterator reduced partial bifurcate abbreviate symbol mask normal assert

Last synced: 4 months ago · JSON representation ·

Repository

Benchmark harness.

Basic Info

Host: GitHub
Owner: stdlib-js
License: apache-2.0
Language: JavaScript
Default Branch: main
Homepage: https://github.com/stdlib-js/stdlib
Size: 2.93 MB

Statistics

Stars: 3
Watchers: 3
Forks: 0
Open Issues: 0
Releases: 0

Topics

bench benchmark harness javascript lib library measure node node-js nodejs perf performance standard stdlib tap

Created over 4 years ago · Last pushed 4 months ago

Metadata Files

Readme Changelog Contributing License Code of conduct Citation Security

We believe in a future in which the web is a preferred environment for numerical computation. To help realize this future, we've built stdlib. stdlib is a standard library, with an emphasis on numerical and scientific computation, written in JavaScript (and C) for execution in browsers and in Node.js.

The library is fully decomposable, being architected in such a way that you can swap out and mix and match APIs and functionality to cater to your exact preferences and use cases.

When you use stdlib, you can be absolutely certain that you are using the most thorough, rigorous, well-written, studied, documented, tested, measured, and high-quality code out there.

To join us in bringing numerical computing to the web, get started by checking us out on GitHub, and please consider financially supporting stdlib. We greatly appreciate your continued support!

Benchmark

Benchmark harness.

## Installation ```bash npm install @stdlib/bench-harness ``` Alternatively, - To load the package in a website via a `script` tag without installation and bundlers, use the [ES Module][es-module] available on the [`esm`][esm-url] branch (see [README][esm-readme]). - If you are using Deno, visit the [`deno`][deno-url] branch (see [README][deno-readme] for usage intructions). - For use in Observable, or in browser/node environments, use the [Universal Module Definition (UMD)][umd] build available on the [`umd`][umd-url] branch (see [README][umd-readme]). - To use as a general utility for the command line, install the corresponding [CLI package][cli-section] globally. The [branches.md][branches-url] file summarizes the available branches and displays a diagram illustrating their relationships. To view installation and usage instructions specific to each branch build, be sure to explicitly navigate to the respective README files on each branch, as linked to above.

## Usage ```javascript var bench = require( '@stdlib/bench-harness' ); ``` #### bench( name\[, options]\[, benchmark] ) Queues a `benchmark` to be run during a subsequent turn of the event loop. After running the `benchmark`, the function outputs benchmark results as Test Anything Protocol ([TAP][tap]) output. ```javascript bench( 'Math.sin', function benchmark( b ) { var x; var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { x = Math.sin( Math.random() ); if ( x !== x ) { b.fail( 'should not return NaN' ); } } b.toc(); if ( x !== x ) { b.fail( 'should not return NaN' ); } b.pass( 'benchmark finished' ); b.end(); }); ``` A `benchmark` function has the following signature: ```javascript function benchmark( b ) { // Benchmark code... } ``` where `b` is a `Benchmark` instance. Synchronous benchmarks should, at minimum, have the following structure: ```javascript function benchmark( b ) { var x; var i; // [1] Start timing: b.tic(); // [2] Loop containing code to time... for ( i = 0; i < b.iterations; i++ ) { // [3] Code to time... // [4] A conditional verifying results to prevent certain compiler optimizations: if ( x !== x ) { b.fail( 'something went wrong!' ); } } // [5] Stop timing: b.toc(); // [6] Another conditional verifying results to prevent certain compiler optimizations: if ( x !== x ) { b.fail( 'something went wrong!' ); } // [7] End the benchmark: b.end(); } ``` Asynchronous benchmarks should have a structure similar to the following: ```javascript function benchmark( b ) { var i = 0; // [1] Start timing: b.tic(); // [2] Asynchronous code to time: return next(); function next( error ) { if ( error ) { return b.fail( error.message ); } i += 1; // [3] Exit condition: if ( i <= b.iterations ) { // Asynchronous task... return; } // [4] Stop timing: b.toc(); // [5] End the benchmark: b.end(); } } ``` For both synchronous and asynchronous benchmarks, calling `b.end()` is **mandatory**, as failing to do so will cause the harness to hang. For example, the following benchmark will **never** complete. ```javascript function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { // Synchronous task... } b.toc(); } ``` **Avoid** making assertions within timed code, as doing so will **significantly** affect raw performance numbers. For example, **avoid** the following: ```javascript function benchmark( b ) { var x; var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { x = Math.sin( Math.random() ); b.equal( x, x, 'does not return NaN' ); // Avoid doing this! } b.toc(); b.equal( x, x, 'does not return NaN' ); // This is fine. b.end(); } ``` Additionally, ensure that all setup code executes **before** calling `b.tic()` and that all cleanup code executes **after** calling `b.toc()`. For example, **avoid** the following: ```javascript function benchmark( b ) { var x; var y; var i; // Start timing: b.tic(); // Setup code: x = new Array( b.iterations ); // Should be before b.tic()! for ( i = 0; i < b.iterations; i++ ) { x[ i ] = Math.random(); } // Code to be timed... for ( i = 0; i < b.iterations; i++ ) { y = Math.sin( x[ i ] ); if ( y !== y ) { b.fail( 'should not return NaN' ); } } // Verify results: b.equal( x, x, 'does not return NaN' ); // Should be after b.toc()! // Stop timing: b.toc(); b.end(); } ``` The function accepts the following `options`: - **iterations**: number of iterations. If `null`, the number of iterations is determined by trying successive powers of `10` until the total time is at least `0.1` seconds. Default: `null`. - **repeats**: number of repeats. Default: `3`. - **timeout**: number of milliseconds before a benchmark is considered failed. Exceeding a timeout does **not**, however, end the benchmark. Ending a long running benchmark requires manual intervention. Default: `300000` (5 minutes). - **skip**: `boolean` indicating whether to skip a benchmark. By default, the harness will automatically determine an iteration number for each benchmark such that a benchmark runs for a length of time sufficient to accurately compute benchmark results. To require a specific number of iterations, set the `iterations` option. ```javascript var opts = { 'iterations': 1e6 }; bench( 'require a specific number of iterations', opts, function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { // Code to be benchmarked... } b.toc(); b.end(); }); ``` To ensure that benchmark results are reproducible, the harness runs each benchmark function multiple times. To specify a repetition number, set the `repeats` option. ```javascript var opts = { 'repeats': 5 }; bench( 'repeat a benchmark multiple times', opts, function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { // Code to be benchmarked... } b.toc(); b.end(); }); ``` To skip a benchmark, set the `skip` option. ```javascript var opts = { 'skip': true }; bench( 'skipped benchmark', opts, function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { // Code to be benchmarked... } b.toc(); b.end(); }); ``` To fail benchmarks which take longer than a specified amount of time to complete, set a `timeout` option (in milliseconds). ```javascript var opts = { 'timeout': 5000 // 5 seconds }; bench( 'async benchmark', opts, function benchmark( b ) { var i = 0; b.tic(); return next(); function next( error ) { if ( error ) { return b.fail( error.message ); } i += 1; if ( i <= b.iterations ) { // Asynchronous task... return; } b.toc(); b.end(); } }); ``` #### bench.onFinish( clbk ) Sets a listener which is invoked once the harness **finishes** running all benchmarks. ```javascript function onFinish() { console.log( 'Done!' ); } bench.onFinish( onFinish ); ``` #### bench.createStream( \[options] ) Returns a results [stream][nodejs-stream]. ```javascript var stdout = require( '@stdlib/streams-node-stdout' ); var stream = bench.createStream(); // Direct all results to `stdout`: stream.pipe( stdout ); var opts = { 'iterations': 1, 'repeats': 1 }; bench( 'beep', opts, function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { b.equal( 3.14, 3.14, 'should be equal' ); } b.toc(); b.end(); }); ``` ```text TAP version 13 # beep ok 1 should be equal --- iterations: 1 elapsed: 0.002985193 rate: 334.98671610177297 ... # 1..1 # total 1 # pass 1 # # ok ``` The results stream can be combined with any [`Writable`][nodejs-writable-stream] stream (e.g., network connection, file, `stdout`, etc). The function accepts the same `options` as [@stdlib/streams/node/transform][@stdlib/streams/node/transform]. For example, by default, the method returns a stream which produces [TAP][tap] output as text. To return an object stream, ```javascript var opts = { 'objectMode': true }; var stream = bench.createStream( opts ); stream.on( 'data', onRow ); function onRow( row ) { console.log( JSON.stringify( row ) ); } opts = { 'iterations': 1, 'repeats': 1 }; bench( 'beep', opts, function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { b.equal( 3.14, 3.14, 'should be equal' ); } b.toc(); b.end(); }); ``` ```text {"type":"benchmark","name":"beep","id":0} {"id":0,"ok":true,"name":"should be equal","operator":"equal","actual":3.14,"expected":3.14,"benchmark":0,"type":"assert"} {"ok":true,"operator":"result","iterations":1,"elapsed":0.00283753,"rate":352.41918147120913,"benchmark":0,"type":"result"} {"benchmark":0,"type":"end"} ``` #### bench.createHarness( \[options]\[, clbk] ) Creates a benchmark harness with a new pending stack and state. ```javascript var harness = bench.createHarness(); harness( 'beep', function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { // Synchronous task... } b.toc(); b.end(); }); ``` To trigger an action when a harness finishes running all benchmarks, provide a callback `function`. ```javascript var harness = bench.createHarness( onFinish ); function onFinish() { harness.close(); } harness( 'beep', function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { // Synchronous task... } b.toc(); b.end(); }); ``` The method accepts the following `options`: - **autoclose**: `boolean` indicating whether to automatically close a harness after running all benchmarks. By default, a `harness` does **not** automatically close. To automatically close a harness once a harness finishes running all benchmarks, set the `autoclose` option to `true`. ```javascript var harness = bench.createHarness({ 'autoclose': true }); harness( 'beep', function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { // Synchronous task... } b.toc(); b.end(); }); ``` * * * ### Harness A `harness` has the following properties and methods... #### harness.createStream( \[options] ) Returns a results [stream][nodejs-stream]. ```javascript var stdout = require( '@stdlib/streams-node-stdout' ); var harness = bench.createHarness(); var stream = harness.createStream(); // Direct all results to `stdout`: stream.pipe( stdout ); var opts = { 'iterations': 1, 'repeats': 1 }; harness( 'beep', opts, function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { b.equal( 'beep', 'beep', 'should be equal' ); } b.toc(); b.end(); }); ``` ```text TAP version 13 # beep ok 1 should be equal --- iterations: 1 elapsed: 0.00166768 rate: 599.6354216636286 ... # 1..1 # total 1 # pass 1 # # ok ``` The method accepts the same `options` as [@stdlib/streams/node/transform][@stdlib/streams/node/transform]. **Note**: benchmarks will **not** run until a destination stream has been created. #### harness.close() Closes a benchmark harness. Any pending benchmarks are cleared from the harness stack. ```javascript var stdout = require( '@stdlib/streams-node-stdout' ); var harness = bench.createHarness(); var stream = harness.createStream(); stream.pipe( stdout ); var opts = { 'iterations': 5, 'repeats': 5 }; harness( 'early close', opts, function benchmark( b ) { var i = 0; b.tic(); setTimeout( next, 0 ); function next() { i += 1; if ( i <= b.iterations ) { b.ok( true, 'should be truthy' ); return setTimeout( next, 10 ); } b.toc(); b.end(); } }); // Early close: setTimeout( onTimeout, 50 ); function onTimeout() { harness.close(); } ``` ```text TAP version 13 # early close ok 1 should be truthy ok 2 should be truthy # WARNING: harness closed before completion. ok 3 should be truthy ok 4 should be truthy ok 5 should be truthy --- iterations: 5 elapsed: 0.05940291 rate: 84.17096064822414 ... ``` **Warning**: a running benchmark may finish **after** closing a harness. #### harness.exit() Forcefully exits a benchmark harness. All pending benchmarks will generate **failing** assertions. ```javascript var stdout = require( '@stdlib/streams-node-stdout' ); var harness = bench.createHarness(); var stream = harness.createStream(); stream.pipe( stdout ); var opts = { 'iterations': 5 }; harness( 'force exit', opts, function benchmark( b ) { var i = 0; b.tic(); return next(); function next() { i += 1; if ( i <= b.iterations ) { b.ok( true, 'should be truthy' ); return setTimeout( next, 10 ); } b.toc(); b.end(); } }); // Forcefully exit: setTimeout( onTimeout, 20 ); function onTimeout() { harness.exit(); } ``` ```text TAP version 13 # force exit ok 1 should be truthy not ok 2 benchmark exited without ending --- operator: fail TODO: include stack ... not ok 3 benchmark exited without ending --- operator: fail TODO: include stack ... ok 4 should be truthy ok 5 should be truthy ok 6 should be truthy ok 7 should be truthy --- iterations: 5 elapsed: 0.061504862 rate: 81.29438612511642 ... ``` **Warning**: a running benchmark may finish **after** exiting a harness. #### harness.exitCode **Read-only** property whose value is the harness exit code. If all benchmarks run successfully (i.e., no failing assertions), the exit code is `0`; otherwise, the exit code is `1`. ```javascript var harness = bench.createHarness(); // Benchmarks only start running when results have a destination: var stream = harness.createStream(); function onFinish() { console.log( harness.exitCode ); // => 1 } var opts = { 'iterations': 1, 'repeats': 1 }; harness( 'exit code', opts, function benchmark( b ) { var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { b.fail( 'failing assertion' ); } b.toc(); b.end(); }); ``` * * * ### Benchmark A `Benchmark` instance has the following properties and methods... #### b.name **Read-only** property whose value is the benchmark `name`. ```javascript var str = b.name; // returns ``` #### b.iterations **Read-only** property whose value is the number of iterations. ```javascript var iter = b.iterations; // returns ``` #### b.tic() Starts a benchmark timer. In order to benchmark code, this method **must always** be called within a `benchmark` function. ```javascript function benchmark( b ) { var x; var i; // Start a timer: b.tic(); for ( i = 0; i < b.iterations; i++ ) { x = Math.sin( Math.random() ); if ( x !== x ) { b.fail( 'should not return NaN' ); } } b.toc(); if ( x !== x ) { b.fail( 'should not return NaN' ); } b.end(); } ``` #### b.toc() Stops a benchmark timer. In order to benchmark code, this method **must always** be called within a `benchmark` function. ```javascript function benchmark( b ) { var x; var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { x = Math.sin( Math.random() ); if ( x !== x ) { b.fail( 'should not return NaN' ); } } // Stop a timer: b.toc(); if ( x !== x ) { b.fail( 'should not return NaN' ); } b.end(); } ``` #### b.end() Explicitly ends a benchmark. In order to benchmark code, this method **must always** be called within a `benchmark` function. ```javascript function benchmark( b ) { var x; var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { x = Math.sin( Math.random() ); if ( x !== x ) { b.fail( 'should not return NaN' ); } } b.toc(); if ( x !== x ) { b.fail( 'should not return NaN' ); } // Explicitly end the benchmark: b.end(); } ``` **Warning**: no assertions should follow a call to `b.end()`. Including assertions after `b.end()` may result in interleaved [TAP][tap] output or an output stream closing before a benchmark executes pending assertions. #### b.comment( msg ) Writes a message without breaking [TAP][tap] output. ```javascript b.comment( 'This is a comment.' ); ``` ```text # This is a comment. ``` To prevent confusing results parsers, **avoid** using comments. Comments are frequently used for demarcating the beginning of a new benchmark run and/or providing diagnostic information. Both of the aforementioned use cases typically fall under the domain of the harness, not the user. #### b.skip( value, msg ) Generates an assertion which will be skipped. ```javascript b.skip( false, 'This is skipped.' ); b.skip( true, 'This is skipped.' ); ``` ```text ok 1 This is skipped. # SKIP ok 2 This is skipped. # SKIP ``` #### b.todo( value, msg ) Generates an assertion which should be implemented. ```javascript b.todo( false, 'This is a todo.' ); b.todo( true, 'This is a todo.' ); ``` ```text not ok 3 This is a todo. # TODO --- operator: todo TODO: include stack ... ok 4 This is a todo. # TODO ``` While `b.todo()` assertions typically fail, they do **not** contribute to the failed assertion count. If a benchmark includes `b.todo()` assertions and no other failing assertions, the benchmark is considered successful. #### b.fail( msg ) Generates a failing assertion. ```javascript b.fail( 'This is a failing assertion.' ); ``` ```text not ok 5 This is a failing assertion. --- operator: fail TODO: include stack ... ``` #### b.pass( msg ) Generates a passing assertion. ```javascript b.pass( 'This is a passing assertion.' ); ``` ```text ok 6 This is a passing assertion. ``` #### b.ok( value\[, msg] ) Asserts that a `value` is truthy. ```javascript b.ok( [] ); ``` ```text ok 7 should be truthy ``` To override the default message, provide a `msg` argument. ```javascript b.ok( true, 'This asserts a value is truthy.' ); b.ok( false, 'This asserts a value is truthy.' ); ``` ```text ok 8 This asserts a value is truthy. not ok 9 This asserts a value is truthy. --- operator: ok TODO: include stack ... ``` #### b.notOk( value\[, msg] ) Asserts that a `value` is falsy. ```javascript b.notOk( null ); ``` ```text ok 10 should be falsy ``` To override the default message, provide a `msg` argument. ```javascript b.notOk( false, 'This asserts a value is falsy.' ); b.notOk( true, 'This asserts a value is falsy.' ); ``` ```text ok 11 This asserts a value is falsy. not ok 12 This asserts a value is falsy. --- operator: notOk TODO: include stack ... ``` #### b.equal( actual, expected\[, msg] ) Asserts that `actual` is **strictly** equal to `expected`. ```javascript var expected = []; var actual = expected; b.equal( actual, expected ); ``` ```text ok 13 should be equal ``` To override the default message, provide a `msg` argument. ```javascript var expected = []; var actual = expected; b.equal( actual, expected, 'This asserts two values are strictly equal.' ); b.equal( 1.0, 2.0, 'This asserts two values are strictly equal.' ); ``` ```text ok 14 This asserts two values are strictly equal. not ok 15 This asserts two values are strictly equal. --- operator: equal TODO: include stack ... ``` #### b.notEqual( actual, expected\[, msg] ) Asserts that `actual` is not **strictly** equal to `expected`. ```javascript var expected = []; var actual = []; b.notEqual( actual, expected ); ``` ```text ok 16 should not be equal ``` To override the default message, provide a `msg` argument. ```javascript var expected = []; var actual = []; b.notEqual( 1.0, 2.0, 'This asserts two values are not equal.' ); b.notEqual( actual, expected, 'This asserts two values are not equal.' ); ``` ```text ok 17 This asserts two values are not equal. not ok 18 This asserts two values are not equal. --- operator: notEqual TODO: include stack ... ``` #### b.deepEqual( actual, expected\[, msg] ) Asserts that `actual` is **deeply** equal to `expected`. ```javascript var expected = { 'a': 'b' }; var actual = { 'a': 'b' }; b.deepEqual( actual, expected ); ``` ```text ok 19 should be deeply equal ``` To override the default message, provide a `msg` argument. ```javascript var expected = { 'a': 'b' }; var actual = { 'a': 'b' }; b.deepEqual( actual, expected, 'This asserts two values are deeply equal.' ); actual.a = 'c'; b.deepEqual( actual, expected, 'This asserts two values are deeply equal.' ); ``` ```text TODO ``` #### b.notDeepEqual( actual, expected\[, msg] ) Asserts that `actual` is not **deeply** equal to `expected`. ```javascript var expected = { 'a': 'b' }; var actual = { 'a': 'c' }; b.notDeepEqual( actual, expected ); ``` ```text ok 22 should not be deeply equal ``` To override the default message, provide a `msg` argument. ```javascript var expected = { 'a': 'b' }; var actual = { 'a': 'c' }; b.notDeepEqual( actual, expected, 'This asserts two values are not deeply equal.' ); actual.a = 'b'; b.notDeepEqual( actual, expected, 'This asserts two values are not deeply equal.' ); ``` ```text TODO ```

## Notes - All benchmark functions execute **serially** in separate turns of the event loop. - All benchmark functions should be added during the **same** turn of the event loop. Otherwise, you will likely encounter race conditions where a benchmark executes and finishes causing a harness to close **before** subsequent benchmarks register. - Similarly, create results streams **before** adding benchmarks to the harness. Otherwise, you will likely miss benchmark results. - If a harness is invoked without providing a `benchmark` function, the benchmark is considered a `todo` and `opts.repeat` is ignored. - All benchmarks are pretested. If a benchmark generates failing assertions or fails to call `b.tic()` and/or `b.toc()` during pretests (even if due to an intermittent failure), a benchmark is **only** run once (i.e., `options.repeats` is ignored). Similarly, if `options.iterations` is `null` and a benchmark fails during iteration number determination, a benchmark is **only** run once and for one iteration. Accordingly, if a benchmark does not run an expected number of repetitions and/or iterations, this behavior is likely attributable to a benchmark failure during pretesting. - All benchmarks must have a `name`. If a `name` is not provided, the harness will throw an `Error`. - While not required, all benchmarks should have a **unique** `name`. Unique names ensure easier identification and assignment of benchmark results. - Uncaught exceptions in benchmark functions are **not** intercepted and will cause the harness to error. - If any one of `b.tic()`, `b.toc()`, or `b.end()` is called more than once within a benchmark, the benchmark will **fail**. - **Always** verify results. Doing so prevents the compiler from performing dead code elimination and other optimization techniques, which would render timing results meaningless. - While many benchmark frameworks calculate various statistics over raw timing results (e.g., mean and standard deviation), do **not** do this. Instead, consider the fastest time an approximate lower bound for how fast an environment can execute benchmark code. Slower times are more likely attributable to other processes interfering with timing accuracy rather than attributable to variability in JavaScript's speed. In which case, the minimum time is most likely the only result of interest. When considering all raw timing results, apply common sense rather than statistics. ### TAP - Results are output in accordance with the Test Anything Protocol ([TAP][tap]) version [13][tap]. - Example [TAP][tap] output: ```text TAP version 13 # Math.hypot --- iterations: 1000000 elapsed: 0.457849215 rate: 2184125.181911691 ... ok 1 benchmark finished # Math.hypot --- iterations: 1000000 elapsed: 0.454676639 rate: 2199365.250432407 ... ok 2 benchmark finished # Math.hypot --- iterations: 1000000 elapsed: 0.472378014 rate: 2116948.652059831 ... ok 3 benchmark finished # hypot --- iterations: 1000000 elapsed: 0.13120811 rate: 7621480.105155086 ... ok 4 benchmark finished # hypot --- iterations: 1000000 elapsed: 0.129308984 rate: 7733414.717727579 ... ok 5 benchmark finished # hypot --- iterations: 1000000 elapsed: 0.12404053 rate: 8061881.064197323 ... ok 6 benchmark finished # 1..6 # total 6 # pass 6 # # ok ``` - For each failing assertion, the harness outputs diagnostic information as [YAML][yaml] blocks. ```text TODO ``` - Timing results are output as [YAML][yaml] blocks. The fields are as follows: - **iterations**: number of iterations. - **elapsed**: total elapsed time beginning with `b.tic()` and ending with `b.toc()` (in seconds). - **rate**: number of operations per second.

## Examples ```javascript var randu = require( '@stdlib/random-base-randu' ); var isnan = require( '@stdlib/math-base-assert-is-nan' ); var sin = require( '@stdlib/math-base-special-sin' ); var bench = require( '@stdlib/bench-harness' ); var opts = { 'iterations': 1e6, 'repeats': 3 }; bench( 'Math.sin', opts, function benchmark( b ) { var x; var y; var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { x = (randu()*100.0) - 50.0; y = Math.sin( x ); if ( y < -1.0 || y > 1.0 ) { b.fail( 'something went wrong!' ); } } b.toc(); if ( isnan( y ) ) { b.fail( 'something went wrong!' ); } b.pass( 'benchmark finished' ); b.end(); }); bench( 'sin', opts, function benchmark( b ) { var x; var y; var i; b.tic(); for ( i = 0; i < b.iterations; i++ ) { x = (randu()*100.0) - 50.0; y = sin( x ); if ( y < -1.0 || y > 1.0 ) { b.fail( 'something went wrong!' ); } } b.toc(); if ( isnan( y ) ) { b.fail( 'something went wrong!' ); } b.pass( 'benchmark finished' ); b.end(); }); ```

## CLI

## Installation To use as a general utility, install the CLI package globally ```bash npm install -g @stdlib/bench-harness-cli ```

### Usage ```text Usage: bench [options] ... Options: -h, --help Print this message. -V, --version Print the package version. -r, --require module Load module before running benchmarks. ```

### Notes - Running benchmark files does **not** require using the command-line interface. For example, to run a single file containing benchmarks, ```bash $ node /path/to/benchmark.js ``` - To load one or more modules **before** running benchmarks, set the `-r` flag one or more times. For example, ```bash $ bench -r foo -r bar 'benchmark/*.js' ``` the modules `foo` and `bar` will both be loaded **before** any benchmarks matching the glob `benchmark/*.js`. The `-r` flag behaves exactly like `require`, and modules are resolved relative to the current working directory. To load local modules, use relative paths. ```bash $ bench -r ./foo/bar.js -r ./beep/boop/bap 'benchmark/*.js' ``` Note that `-r` modules are loaded **before** running benchmarks regardless of order. Hence, ```bash $ bench -r foo -r bar 'benchmark/*.js' ``` and ```bash $ bench -r foo 'benchmark/*.js' -r bar ``` behave the same. Depending on the preloaded module, a module may support parameterization via environment variables, command-line options, and/or configuration files. - To perform shell expansion on systems supporting globbing, do not quote provided globs. ```bash $ bench benchmark/*.js ``` Beware, however, that globbing via shell expansion may result in shell argument lists which exceed length limits. To prevent shell expansion, wrap globs in quotes. ```bash $ bench 'benchmark/*.js' $ bench "benchmark/*.js" ```

### Examples ```bash $ bench ./examples/index.js ``` will generate [TAP][tap] output similar to the following ```text TAP version 13 # Math.sin --- iterations: 1000000 elapsed: 0.107631765 rate: 9290937.484858675 ... ok 1 benchmark finished # Math.sin --- iterations: 1000000 elapsed: 0.100319363 rate: 9968165.368035682 ... ok 2 benchmark finished # Math.sin --- iterations: 1000000 elapsed: 0.095116262 rate: 10513449.31952856 ... ok 3 benchmark finished # sin --- iterations: 1000000 elapsed: 0.173696195 rate: 5757178.503536016 ... ok 4 benchmark finished # sin --- iterations: 1000000 elapsed: 0.158544701 rate: 6307369.42762912 ... ok 5 benchmark finished # sin --- iterations: 1000000 elapsed: 0.157709895 rate: 6340756.234730865 ... ok 6 benchmark finished # 1..6 # total 6 # pass 6 # # ok ```

## References - Chen, Jiahao, and Jarrett Revels. 2016. "Robust benchmarking in noisy environments." _CoRR_ abs/1608.04295 (August). .

* * * ## Notice This package is part of [stdlib][stdlib], a standard library for JavaScript and Node.js, with an emphasis on numerical and scientific computing. The library provides a collection of robust, high performance libraries for mathematics, statistics, streams, utilities, and more. For more information on the project, filing bug reports and feature requests, and guidance on how to develop [stdlib][stdlib], see the main project [repository][stdlib]. #### Community [![Chat][chat-image]][chat-url] --- ## License See [LICENSE][stdlib-license]. ## Copyright Copyright © 2016-2025. The Stdlib [Authors][stdlib-authors].

Owner

Name: stdlib
Login: stdlib-js
Kind: organization

Website: https://stdlib.io
Twitter: stdlibjs
Repositories: 3,692
Profile: https://github.com/stdlib-js

Standard library for JavaScript.

Citation (CITATION.cff)

cff-version: 1.2.0
title: stdlib
message: >-
  If you use this software, please cite it using the
  metadata from this file.

type: software

authors:
  - name: The Stdlib Authors
    url: https://github.com/stdlib-js/stdlib/graphs/contributors

repository-code: https://github.com/stdlib-js/stdlib
url: https://stdlib.io

abstract: |
  Standard library for JavaScript and Node.js.

keywords:
  - JavaScript
  - Node.js
  - TypeScript
  - standard library
  - scientific computing
  - numerical computing
  - statistical computing

license: Apache-2.0 AND BSL-1.0

date-released: 2016

GitHub Events

Total

Push event: 46

Last Year

Push event: 46

Committers

Last synced: almost 2 years ago

All Time

Total Commits: 71
Total Committers: 2
Avg Commits per committer: 35.5
Development Distribution Score (DDS): 0.028

Past Year

Commits: 22
Committers: 2
Avg Commits per committer: 11.0
Development Distribution Score (DDS): 0.045

Top Committers

Name	Email	Commits
stdlib-bot	n**y@s**o	69
Philipp Burckhardt	p**t@o**m	2

Committer Domains (Top 20 + Academic)

stdlib.io: 1

Issues and Pull Requests

Last synced: 4 months ago

All Time

Total issues: 0
Total pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Total issue authors: 0
Total pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 0
Pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Issue authors: 0
Pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

Total packages: 2
Total downloads:
- npm 10,541 last-month

Total dependent packages: 1
(may contain duplicates)
Total dependent repositories: 0
(may contain duplicates)
Total versions: 16
Total maintainers: 4

npmjs.org: @stdlib/bench-harness

Benchmark harness.

Homepage: https://stdlib.io
License: Apache-2.0
Latest release: 0.2.2
published over 1 year ago

Versions: 13
Dependent Packages: 1
Dependent Repositories: 0
Downloads: 10,540 Last month

Rankings

Downloads: 13.5%

Dependent packages count: 16.2%

Stargazers count: 16.8%

Forks count: 17.4%

Average: 17.9%

Dependent repos count: 25.3%

Maintainers (4)

planeshifter rreusser stdlib-bot kgryte

Funding

type: opencollective
url: https://opencollective.com/stdlib

Last synced: 4 months ago

npmjs.org: @stdlib/bench-harness-cli

Benchmark harness.

Homepage: https://stdlib.io
License: Apache-2.0
Latest release: 0.2.2
published over 1 year ago

Versions: 3
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 1 Last month

Rankings

Dependent repos count: 33.9%

Average: 41.2%

Dependent packages count: 48.4%

Maintainers (4)

planeshifter rreusser stdlib-bot kgryte

Funding

type: opencollective
url: https://opencollective.com/stdlib

Last synced: 4 months ago

Dependencies

package.json npm

@stdlib/assert-contains ^0.0.x development
@stdlib/assert-is-regexp ^0.0.x development
@stdlib/assert-is-windows ^0.0.x development
@stdlib/bench ^0.0.x development
@stdlib/buffer-from-string ^0.0.x development
@stdlib/math-base-assert-is-nan ^0.0.x development
@stdlib/math-base-special-cos ^0.0.x development
@stdlib/math-base-special-sin ^0.0.x development
@stdlib/process-exec-path ^0.0.x development
@stdlib/random-base-randu ^0.0.x development
istanbul ^0.4.1 development
proxyquire ^2.0.0 development
tap-spec 5.x.x development
tape git+https://github.com/kgryte/tape.git#fix/globby development
@stdlib/assert-has-own-property ^0.0.x
@stdlib/assert-is-boolean ^0.0.x
@stdlib/assert-is-browser ^0.0.x
@stdlib/assert-is-function ^0.0.x
@stdlib/assert-is-node-writable-stream-like ^0.0.x
@stdlib/assert-is-null ^0.0.x
@stdlib/assert-is-plain-object ^0.0.x
@stdlib/assert-is-positive-integer ^0.0.x
@stdlib/assert-is-string ^0.0.x
@stdlib/cli-ctor ^0.0.x
@stdlib/fs-read-file ^0.0.x
@stdlib/process-cwd ^0.0.x
@stdlib/regexp-eol ^0.0.x
@stdlib/streams-node-transform ^0.0.x
@stdlib/string-format ^0.0.x
@stdlib/string-from-code-point ^0.0.x
@stdlib/string-replace ^0.0.x
@stdlib/string-trim ^0.0.x
@stdlib/time-tic ^0.0.x
@stdlib/time-toc ^0.0.x
@stdlib/utils-copy ^0.0.x
@stdlib/utils-define-nonenumerable-read-only-accessor ^0.0.x
@stdlib/utils-define-nonenumerable-read-only-property ^0.0.x
@stdlib/utils-define-property ^0.0.x
@stdlib/utils-inherit ^0.0.x
@stdlib/utils-noop ^0.0.x
@stdlib/utils-omit ^0.0.x
@stdlib/utils-pick ^0.0.x
glob ^7.0.5
resolve ^1.1.7

.github/workflows/benchmark.yml actions

actions/checkout v3 composite
actions/setup-node v3 composite

.github/workflows/cancel.yml actions

styfle/cancel-workflow-action 0.11.0 composite

.github/workflows/close_pull_requests.yml actions

superbrothers/close-pull-request v3 composite

.github/workflows/examples.yml actions

actions/checkout v3 composite
actions/setup-node v3 composite

.github/workflows/npm_downloads.yml actions

actions/checkout v3 composite
actions/setup-node v3 composite
actions/upload-artifact v3 composite
distributhor/workflow-webhook v3 composite

.github/workflows/productionize.yml actions

act10ns/slack v1 composite
actions/checkout v3 composite
actions/setup-node v3 composite
stdlib-js/bundle-action main composite
stdlib-js/transform-errors-action main composite

.github/workflows/publish.yml actions

JS-DevTools/npm-publish v1 composite
act10ns/slack v1 composite
actions/checkout v3 composite
actions/setup-node v3 composite
styfle/cancel-workflow-action 0.11.0 composite

.github/workflows/test.yml actions

act10ns/slack v1 composite
actions/checkout v3 composite
actions/setup-node v3 composite

.github/workflows/test_bundles.yml actions

act10ns/slack v1 composite
actions/checkout v3 composite
actions/setup-node v3 composite
denoland/setup-deno v1 composite

.github/workflows/test_coverage.yml actions

act10ns/slack v1 composite
actions/checkout v3 composite
actions/setup-node v3 composite
codecov/codecov-action v3 composite
distributhor/workflow-webhook v3 composite

.github/workflows/test_install.yml actions

act10ns/slack v1 composite
actions/checkout v3 composite
actions/setup-node v3 composite

.github/workflows/publish_cli.yml actions

JS-DevTools/npm-publish fe72237be0920f7a0cafd6a966c9b929c9466e9b composite
act10ns/slack ed1309ab9862e57e9e583e51c7889486b9a00b0f composite
actions/checkout 8ade135a41bc03ea155e62e844d188df1ea18608 composite
actions/setup-node 5e21ff4d9bc1a8cf6de233a3057d20ec6b3fb69d composite
styfle/cancel-workflow-action b173b6ec0100793626c2d9e6b90435061f4fc3e5 composite

@stdlib/bench-harness

Science Score: 54.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

Benchmark

Owner

Citation (CITATION.cff)

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

npmjs.org: @stdlib/bench-harness

Rankings

Maintainers (4)

Funding

npmjs.org: @stdlib/bench-harness-cli

Rankings

Maintainers (4)

Funding

Dependencies