Dagger
A framework for out-of-core and parallel execution
Science Score: 54.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
4 of 59 committers (6.8%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (15.1%) to scientific vocabulary
Keywords from Contributors
Repository
A framework for out-of-core and parallel execution
Basic Info
Statistics
- Stars: 680
- Watchers: 24
- Forks: 76
- Open Issues: 116
- Releases: 88
Metadata Files
README.md
A framework for out-of-core and parallel computing
| Documentation | Build Status |
|:---------------------------------------:|:---------------------------------------:|
| 
| 
|
At the core of Dagger.jl is a scheduler heavily inspired by Dask. It can run computations represented as directed-acyclic-graphs (DAGs) efficiently on many Julia worker processes and threads, as well as GPUs via DaggerGPU.jl.
The *DTable** has been moved out of this repository. You can now find it here.*
Installation
Dagger.jl can be installed using the Julia package manager. Enter the Pkg REPL mode by typing "]" in the Julia REPL and then run:
julia
pkg> add Dagger
Or, equivalently, install Dagger via the Pkg API:
julia
julia> import Pkg; Pkg.add("Dagger")
Usage
Once installed, the Dagger package can be loaded with using Dagger, or if
you want to use Dagger for distributed computing, it can be loaded as:
julia
using Distributed; addprocs() # Add one Julia worker per CPU core
using Dagger
You can run the following example to see how Dagger exposes easy parallelism:
```julia
This runs first:
a = Dagger.@spawn rand(100, 100)
These run in parallel:
b = Dagger.@spawn sum(a) c = Dagger.@spawn prod(a)
Finally, this runs:
wait(Dagger.@spawn println("b: ", b, ", c: ", c)) ```
Use Cases
Dagger can support a variety of use cases that benefit from easy, automatic parallelism, such as:
This isn't an exhaustive list of the use cases that Dagger supports. There are more examples in the docs, and more use cases examples are welcome (just file an issue or PR).
Contributing Guide
Please see the roadmap for missing features or known bugs:
Other resources:
Contributions are encouraged.
There are several ways to contribute to our project:
Reporting Bugs: If you find a bug, please open an issue and describe the problem. Make sure to include steps to reproduce the issue and any error messages you receive regarding that issue.
Fixing Bugs: If you'd like to fix a bug, please create a pull request with your changes. Make sure to include a description of the problem and how your changes will address it.
Additional examples and documentation improvements are also very welcome.
Resources
List of recommended Dagger.jl resources:
- Docs
- Videos
- Distributed Computing with Dagger.jl
- Easy, Featureful Parallelism with Dagger.jl
- Easier parallel Julia workflow with Dagger.jl
- Dagger.jl Development and Roadmap
Help and Discussion
For help and discussion, we suggest asking in the following places:
Julia Discourse and on the Julia Slack in the #dagger channel.
References
bibtex
@inproceedings{alomairy2024dynamic,
title={Dynamic Task Scheduling with Data Dependency Awareness Using Julia},
author={Alomairy, Rabab and Tome, Felipe and Samaroo, Julian and Edelman, Alan},
booktitle={2024 IEEE High Performance Extreme Computing Conference (HPEC)},
pages={1--7},
year={2024},
organization={IEEE}
}
bibtex
@article{dagger2,
title={Efficient Dynamic Task Scheduling in Heterogeneous Environments with Julia},
author={Samaroo, Julian and Alomairy, Rabab and and Giordano, Mose and Edelman, Alan},
year={2024},
publisher={MIT Open Access Articles}
}
Acknowledgements
We thank DARPA, Intel, and the NIH for supporting this work at MIT.
Owner
- Name: JuliaParallel
- Login: JuliaParallel
- Kind: organization
- Website: https://julialang.org
- Repositories: 36
- Profile: https://github.com/JuliaParallel
Parallel programming in Julia
Citation (CITATION.cff)
cff-version: 1.2.0 message: "If you use this software, please cite it as below." authors: - family-names: "Samaroo" given-names: "Julian" orcid: "https://orcid.org/0009-0003-9623-0512" - family-names: "Alomairy" given-names: "Rabab" orcid: "https://orcid.org/0000-0001-9911-6094" - family-names: "de Alcântara Tomé" given-names: "Felipe" orcid: "https://orcid.org/0000-0002-1448-9340" - family-names: "Wrigley" given-names: "James" orcid: "https://orcid.org/0009-0003-6525-7413" title: "Dagger.jl" version: 0.18.12 date-released: 2024-06-25 url: "https://github.com/JuliaParallel/Dagger.jl"
GitHub Events
Total
- Create event: 58
- Commit comment event: 15
- Issues event: 14
- Release event: 6
- Watch event: 50
- Delete event: 25
- Issue comment event: 48
- Push event: 211
- Pull request review event: 15
- Pull request review comment event: 68
- Pull request event: 78
- Fork event: 11
Last Year
- Create event: 58
- Commit comment event: 15
- Issues event: 14
- Release event: 6
- Watch event: 50
- Delete event: 25
- Issue comment event: 48
- Push event: 211
- Pull request review event: 15
- Pull request review comment event: 68
- Pull request event: 78
- Fork event: 11
Committers
Last synced: 9 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Julian P Samaroo | j****o@j****e | 520 |
| Shashi Gowda | s****1@g****m | 203 |
| Shashi Gowda | s****i@j****m | 103 |
| krynju | g****n@g****m | 22 |
| JamesWrigley | j****s@p****g | 20 |
| tan | t****m@g****m | 16 |
| Mateusz Jakub Fila | m****a@c****h | 9 |
| Wolf Thomsen | g****t@w****t | 8 |
| CompatHelper Julia | c****y@j****g | 7 |
| Kenta Sato | b****5@g****m | 7 |
| rabab53 | r****y@k****a | 7 |
| Martijn Visser | m****r@g****m | 6 |
| Pranav | p****t@g****m | 6 |
| Viral B. Shah | v****l@m****g | 6 |
| dependabot[bot] | 4****] | 5 |
| DrChainsaw | C****y@g****m | 5 |
| Jeff Bezanson | j****n@g****m | 5 |
| Chris de Graaf | me@c****v | 5 |
| Matt Bhagat-Conway | m****t@i****g | 4 |
| Andreas Noack | a****n@g****m | 4 |
| Tim Besard | t****d@g****m | 3 |
| Rohit Varkey Thankachan | r****y@g****m | 3 |
| Dilum Aluthge | d****m@a****m | 3 |
| Felipe de Alcântara Tomé | 6****e | 2 |
| pszufe | p****e@g****m | 2 |
| femtocleaner[bot] | f****] | 2 |
| ScottPJones | s****s@a****u | 2 |
| Elliot Saba | s****t@g****m | 2 |
| n0rbed | y****9@g****m | 2 |
| Robert K Samuel | 3****n | 1 |
| and 29 more... | ||
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 99
- Total pull requests: 273
- Average time to close issues: 3 months
- Average time to close pull requests: 28 days
- Total issue authors: 42
- Total pull request authors: 33
- Average comments per issue: 2.72
- Average comments per pull request: 0.71
- Merged pull requests: 198
- Bot issues: 0
- Bot pull requests: 15
Past Year
- Issues: 12
- Pull requests: 89
- Average time to close issues: 4 months
- Average time to close pull requests: 9 days
- Issue authors: 8
- Pull request authors: 12
- Average comments per issue: 0.33
- Average comments per pull request: 0.28
- Merged pull requests: 47
- Bot issues: 0
- Bot pull requests: 4
Top Authors
Issue Authors
- jpsamaroo (25)
- schlichtanders (8)
- StevenWhitaker (8)
- m-fila (6)
- krynju (4)
- DrChainsaw (4)
- eschnett (4)
- Roh-codeur (3)
- wolthom (2)
- islent (2)
- AkhilAkkapelli (1)
- mrufsvold (1)
- dbedi3311 (1)
- ederag (1)
- josia-pool (1)
Pull Request Authors
- jpsamaroo (202)
- JamesWrigley (37)
- github-actions[bot] (16)
- AkhilAkkapelli (13)
- m-fila (11)
- Rabab53 (10)
- krynju (9)
- fda-tome (8)
- dependabot[bot] (6)
- yanzin00 (4)
- wolthom (4)
- Madhupatel08 (4)
- n0rbed (4)
- pszufe (4)
- johnmaxrin (3)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 3
-
Total downloads:
- julia 268 total
-
Total dependent packages: 17
(may contain duplicates) -
Total dependent repositories: 1
(may contain duplicates) - Total versions: 76
juliahub.com: Dagger
A framework for out-of-core and parallel execution
- Documentation: https://docs.juliahub.com/General/Dagger/stable/
- License: MIT
-
Latest release: 0.19.1
published 7 months ago
Rankings
juliahub.com: TimespanLogging
A framework for out-of-core and parallel execution
- Documentation: https://docs.juliahub.com/General/TimespanLogging/stable/
- License: MIT
-
Latest release: 0.1.1
published 10 months ago
Rankings
juliahub.com: DaggerWebDash
- Documentation: https://docs.juliahub.com/General/DaggerWebDash/stable/
- License: MIT
-
Latest release: 0.1.4
published 6 months ago