data.table.threads
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.5%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: Anirban166
- License: other
- Language: R
- Default Branch: main
- Homepage: https://CRAN.R-project.org/package=data.table.threads
- Size: 188 KB
Statistics
- Stars: 3
- Watchers: 1
- Forks: 1
- Open Issues: 1
- Releases: 0
Metadata Files
README.md
Installation
Fetch from CRAN using:
r
install.packages("data.table.threads")
or use the latest (developmental) version from here:
r
if(!require(remotes)) install.packages("remotes"); remotes::install_github("Anirban166/data.table.threads")
r
if(!require(devtools)) install.packages("devtools"); devtools::install_github("Anirban166/data.table.threads")
Usage
findOptimalThreadCount(rowCount, columnCount, ...) is the go-to function that runs a set of predefined benchmarks for various data.table functions that are parallelizable, across varying thread counts (iteratively from one to the highest number available as per the user's system). It involves computation to find the optimal/ideal speedup and thread count for each function. It returns a data.table object of a custom class (print and plot methods have been provided), which contains the optimal thread count for each function. It also provides plot data (consisting of speedup trends and key points) as attributes.
```r
(benchmarks <- data.table.threads::findOptimalThreadCount(1e7, 10))
Function Thread count Fastest median runtime (ms)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
forder 8 102.786918
GForce_sum 4 15.656137
subsetting 5 54.542259
frollmean 5 23.541975
fcoalesce 10 6.830682
between 6 23.160429
fifelse 8 19.139230
nafill 4 7.095849
CJ 4 3.280164
``
The output here is a table which shows the fastest runtime (median value in milliseconds) for each applicabledata.table` function along with the corresponding thread count that achieved it.
Plotting this object would generate a plot that shows the ideal and measured speedup trends for each routine:
r
plot(benchmarkData)
If the user wants to factor in a specified speedup efficiency, they can use the function addRecommendedEfficiency to add a speedup line (with a slope configured by input argument efficiencyFactor; default value is 0.5, or 50% efficiency) along with a point representing the recommended thread count which stems from the highest intersection between this line (of specified thread-use efficiency) and measured speedup data for each function:
r
benchmarks_r <- addRecommendedEfficiency(benchmarks, recommendedEfficiency = 0.4)
plot(benchmarks_r)
In both cases (with or without the addition of recommended efficiency), the generated plot delineates the speedup across multiple threads (from 1 to the number of threads available in the user's system; 10 in my case here) for each function.
setThreadCount(benchmarks, functionName, efficiencyFactor) can then be used to set the thread count based on the observed results for a user-specified function and efficiency value (of the range [0, 1]) for the speedup:
```r
setThreadCount(benchmarks_r, functionName = "forder", efficiencyFactor = 0.6, verbose = TRUE)
The number of threads that data.table will use has been set to 2, based on an efficiency factor of 0.6 for data.table::forder() based on the performed benchmarks.
getDTthreads()
[1] 2
```
When using findOptimalThreadCount(), users can also replace the predefined benchmarks with their own expressions by providing a list of custom functions as the benchmarksList argument, enabling evaluation tailored to their specific use cases. On top of that, they can also specify their own data.table via the customDT argument to have the functions they define operate on it instead of the default matrix-based data.table that makes use of rowCount and colCount.
For instance, here is an example: ```r
Derived from https://github.com/Rdatatable/data.table/issues/4294
NN = 1e5 DT = data.table(grp1 = as.character(rep(1:(NN / 4), each = 4)), grp2 = sample(5000L, NN, TRUE), V = rpois(NN, 10)) findOptimalThreadCount(benchmarksList = list(test4294 = function(dt) dt[, log(sum(V)), by = grp1]), customDT = DT)
Optimal efficiency is observed for a single thread in this case since the code includes creation of many small groups where the computation for each of them is relatively lightweight to see improvement in performance outweighing the overhead from multi-threading.
```
Owner
- Name: Ani
- Login: Anirban166
- Kind: user
- Location: 0x(7F.0.0.1), CA/AZ
- Company: Future Gadget Lab
- Website: https://anirban166.github.io/
- Repositories: 63
- Profile: https://github.com/Anirban166
GitHub Events
Total
- Issues event: 6
- Watch event: 1
- Issue comment event: 19
- Push event: 27
- Pull request review event: 4
- Pull request review comment event: 3
- Pull request event: 3
- Create event: 2
Last Year
- Issues event: 6
- Watch event: 1
- Issue comment event: 19
- Push event: 27
- Pull request review event: 4
- Pull request review comment event: 3
- Pull request event: 3
- Create event: 2
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Anirban | b****6@g****m | 195 |
| Toby Dylan Hocking | t****g@r****g | 2 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 11 months ago
All Time
- Total issues: 16
- Total pull requests: 11
- Average time to close issues: 8 days
- Average time to close pull requests: 6 days
- Total issue authors: 2
- Total pull request authors: 2
- Average comments per issue: 3.94
- Average comments per pull request: 1.82
- Merged pull requests: 11
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 11
- Pull requests: 8
- Average time to close issues: 4 days
- Average time to close pull requests: 1 day
- Issue authors: 2
- Pull request authors: 2
- Average comments per issue: 4.27
- Average comments per pull request: 0.5
- Merged pull requests: 8
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- Anirban166 (10)
- tdhock (9)
Pull Request Authors
- Anirban166 (17)
- tdhock (4)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- cran 623 last-month
- Total dependent packages: 0
- Total dependent repositories: 0
- Total versions: 2
- Total maintainers: 1
cran.r-project.org: data.table.threads
Analyze Multi-Threading Performance for 'data.table' Functions
- Homepage: https://github.com/Anirban166/data.table.threads
- Documentation: http://cran.r-project.org/web/packages/data.table.threads/data.table.threads.pdf
- License: MIT + file LICENSE
-
Latest release: 1.0.1
published over 1 year ago
Rankings
Maintainers (1)
Dependencies
- data.table * imports
- ggplot2 * imports
- microbenchmark * imports
- testthat >= 2.1.0 imports