Releases | Open Source Science

Dagger - v0.19.1

Dagger v0.19.1

Merged pull requests: - scopes/Sch: Add optimized check for proc in scope (#640) (@jpsamaroo) - Add support for @inbounds via Base.@boundscheck in abstract array types (#643) (@pszufe) - CI: Remove DTables reverse CI (#644) (@jpsamaroo) - Sch: Quietly fail to put do_tasks exception (#645) (@jpsamaroo) - change div to cld to avoid DivideError in FFT (#648) (@islent)

Closed issues: - 3D FFT ConcurrencyViolationError (#646) - 3D FFT gets DivideError: integer division error (#647)

- Julia
Published by github-actions[bot] 7 months ago

Dagger - v0.19.0

Dagger v0.19.0

Diff since v0.18.17

Breaking changes

The result of delayed(...)(...) can no longer be evaluated twice
ThunkOptions has been replaced with Options
Arbitrary options cannot be propagated anymore
The Processor component of Chunk, and the processor option, are no longer relevant to scheduling

Merged pull requests: - Various performance and memory optimizations (#558) (@jpsamaroo) - datadeps: Add at-stencil helper (#564) (@jpsamaroo) - Add task affinity support with computescope and resultscope in Dagger.jl's @spawn macro (#610) (@AkhilAkkapelli) - logging: Default disable metrics, enable alltaskdeps (#626) (@jpsamaroo) - Add view Support for Efficient DArray Chunk Slicing (#627) (@AkhilAkkapelli) - DArray: Add FFT implementation for 1D/2D/3D (#629) (@jpsamaroo) - Enhance DArray Distribution with Processor Assignment (#630) (@AkhilAkkapelli) - CompatHelper: bump compat for AMDGPU in [weakdeps] to 2, (keep existing compat) (#637) (@github-actions[bot])

Closed issues: - Memory leak when copying data from one worker to another (#547)

- Julia
Published by github-actions[bot] 8 months ago

Dagger - v0.18.17

Dagger v0.18.17

Diff since v0.18.16

Merged pull requests: - tests: Test showlogs method (#601) (@jpsamaroo) - docs: Document Dagger.setdistributedpackage! (#602) (@jpsamaroo) - docs: Make note about required packages for visualization (#603) (@jpsamaroo) - JSON3Ext: Deprecate Dagger.renderlogs for :chrome_trace (#604) (@jpsamaroo) - docs: Adjust wording and headers for viz (#605) (@jpsamaroo) - docs: Clarify limitations of Datadeps (#606) (@jpsamaroo) - docs: Define Datadeps example functions with @everywhere (#607) (@jpsamaroo) - datadeps: Support field access in Deps (#608) (@jpsamaroo) - datadeps: Support inner task scopes (#609) (@jpsamaroo) - datadeps: Reduce dynamic dispatch with wrappers (#612) (@jpsamaroo) - CI: Add Buildkite test jobs for Julia 1.10 and 1.11 (#613) (@jpsamaroo) - docs: Simplify logging basics (#614) (@jpsamaroo) - docs: Move Options subsection in task-spawning.md (#615) (@jpsamaroo) - Add documentation for different task spawning syntaxes (#616) (@jpsamaroo) - docs: Document move and move! functions (#617) (@jpsamaroo) - stencils: Add tests and documentation (#619) (@jpsamaroo) - Fix DArray setindex! scope, and support threads=: in Dagger.scope() (#620) (@jpsamaroo) - Move GPU support into Dagger (#623) (@jpsamaroo)

Closed issues: - 'No TID' warnings when shard is used (#571) - Error when using Dagger in Jupyter Notebook / Google Colab (#594) - Error with docs example on nested loops (#595) - Dagger.workers() do not include DistributedNext workers ? (#600)

- Julia
Published by github-actions[bot] 8 months ago

Dagger - v0.18.16

Dagger v0.18.16

Diff since v0.18.15

Merged pull requests: - precompile: Switch task cleanup assert to warn (#599) (@jpsamaroo)

Closed issues: - Dagger won't precompile in Julia 1.12-beta3 (#598)

- Julia
Published by github-actions[bot] 9 months ago

Dagger - v0.18.15

Dagger v0.18.15

Diff since v0.18.14

Merged pull requests: - Add streaming API (#463) (@jpsamaroo) - datadeps: Don't skip copy on no writedep (#580) (@jpsamaroo) - streaming: Add DAG teardown option (#584) (@jpsamaroo) - parser: Support setindex (#585) (@jpsamaroo) - Hash USER environment variable during system_uuid fallback (#587) (@nHackel) - docs: writes to parameter passed to Out(), doesnt read from it (#590) (@n0rbed) - x is uninitialized in docs (#593) (@n0rbed)

Closed issues: - Bump TimespanLogging and register new version (#597)

- Julia
Published by github-actions[bot] 10 months ago

Dagger - v0.18.14

Dagger v0.18.14

Diff since v0.18.13

Merged pull requests: - Initial support for robustly migrating streaming tasks (#568) (@JamesWrigley) - CompatHelper: add new compat entry for PythonCall in [weakdeps] at version 0.9, (keep existing compat) (#574) (@github-actions[bot]) - parser: Re-support broadcast (#575) (@jpsamaroo) - Use [sources] to specify local deps for documentation (#576) (@m-fila) - CompatHelper: bump compat for Colors in [weakdeps] to 0.13, (keep existing compat) (#577) (@github-actions[bot]) - Update CI jobs (#578) (@JamesWrigley) - Add support for DistributedNext (#582) (@JamesWrigley)

Closed issues: - Dagger documentation does not build locally (#552)

- Julia
Published by github-actions[bot] about 1 year ago

Dagger - v0.18.13

Dagger v0.18.13

Diff since v0.18.12

Merged pull requests: - Add render/showlogs to chrometrace (#540) (@m-fila) - Include datadeps in API documentation (#541) (@m-fila) - CompatHelper: add new compat entry for JSON3 in [weakdeps] at version 1, (keep existing compat) (#542) (@github-actions[bot]) - Add a Features/Roadmap document (#544) (@jpsamaroo) - Add rand! implementation (#546) (@pszufe) - add documentation for Task / Thread occupancy (#549) (@wolthom) - Warn in docs about sync on inner loops (#550) (@quildtide) - docs: Create favicon and add it to the documentation (#551) (@wolthom) - @spawn: Properly rethrow in @sync (#553) (@jpsamaroo) - More Thunk to DTask renames (#554) (@jpsamaroo) - Add fetch_all recursive helper (#555) (@jpsamaroo) - @spawn: Support NamedTuple construction and access (#556) (@jpsamaroo) - Add task/scheduler cancellation API (#557) (@jpsamaroo) - Replace all uses of @async with Threads.@spawn (#559) (@JamesWrigley) - Task names, and various GraphVizExt and PlotsExt improvements (#560) (@jpsamaroo) - Update README.md (#562) (@Rabab53) - LU implementation for DArray (#565) (@jpsamaroo) - Make Dagger.cancel!() handle a shutting down/shutdown scheduler (#566) (@JamesWrigley) - use user given task names in chrome trace (#570) (@m-fila) - Add Python integration (#572) (@jpsamaroo)

Closed issues: - Assignable task names for logging purposes (#545) - Parallelization across threads does not appear to work correctly (#548)

- Julia
Published by github-actions[bot] over 1 year ago

Dagger - v0.18.12

Dagger v0.18.12

Diff since v0.18.11

Merged pull requests: - DArray: Fix DMatrix show elements (#530) (@jpsamaroo) - Dead worker handling (#532) (@JamesWrigley) - parser: Assorted improvements (#533) (@jpsamaroo) - aliasing: Add optimized will_alias for views (#534) (@jpsamaroo) - DArray: Make allocations configurable via Processor (#535) (@jpsamaroo) - Sch: Halt on Julia exit (#537) (@jpsamaroo) - Fix typos in documentation, tests and benchmarks (#538) (@m-fila) - Minor fixes to Datadeps for GPUs (#539) (@jpsamaroo)

Closed issues: - Dagger's syntax when calling functions with kwards does not match Julia syntax (#421) - BUG @spawn macro does not support do syntax (#479) - @spawn does not work with nested function call (#480) - Adding processes and using eager API produces warnings about workers dying (#536)

- Julia
Published by github-actions[bot] over 1 year ago

Dagger - v0.18.11

Dagger v0.18.11

Diff since v0.18.10

Merged pull requests: - Add support for limiting the evaluations of a streaming DAG (#519) (@JamesWrigley) - CompatHelper: add new compat entry for GraphViz in [weakdeps] at version 0.2, (keep existing compat) (#520) (@github-actions[bot]) - CompatHelper: add new compat entry for DataFrames in [weakdeps] at version 1, (keep existing compat) (#521) (@github-actions[bot]) - CompatHelper: add new compat entry for Colors in [weakdeps] at version 0.12, (keep existing compat) (#522) (@github-actions[bot]) - CompatHelper: add new compat entry for Plots in [weakdeps] at version 1, (keep existing compat) (#523) (@github-actions[bot]) - DArray: Fix broken indexing caching (#525) (@jpsamaroo) - tests: Fix assumption that rand won't return 0 (#526) (@jpsamaroo) - delayed: Make depwarn less noisy (#527) (@jpsamaroo) - DArray: Improve 0-dim support (#528) (@jpsamaroo)

- Julia
Published by github-actions[bot] almost 2 years ago

Dagger - v0.18.10

Dagger v0.18.10

Diff since v0.18.9

Merged pull requests: - Cleanup imports with ExplicitImports.jl (#504) (@JamesWrigley) - DArray: Add automatic partitioning and show array elements (#505) (@jpsamaroo) - build(deps): bump julia-actions/cache from 1 to 2 (#508) (@dependabot[bot]) - Add a section to the documentation about code loading (#510) (@JamesWrigley) - Add type restrictions to Base.isless() methods to fix invalidations (#511) (@JamesWrigley) - logging/viz: Some fixes, and more docstrings (#513) (@jpsamaroo) - Add option to the tests for setting the number of workers (#515) (@JamesWrigley) - Test fixes (#516) (@JamesWrigley) - Rename EagerThunk to DTask (#518) (@jpsamaroo) - Datadeps: Add aliasing awareness and GPU support (#524) (@jpsamaroo)

Closed issues: - function show_plan not working (#433) - Cannot precompile Dagger on Julia 1.11.0-beta1 (#503) - KeyError: key Dagger not found (#509)

- Julia
Published by github-actions[bot] almost 2 years ago

Dagger - v0.18.9

Dagger v0.18.9

Diff since v0.18.8

Merged pull requests: - logging: Make IDs consistent, add helpers for logging and visualization (#468) (@jpsamaroo) - @shard: Construct shards in parallel (#481) (@jpsamaroo) - Reference Dagger.EAGERTHUNKSTREAMS explicitly (#484) (@JamesWrigley) - DArray: Fixes and improvements for mapreduce operations (#485) (@jpsamaroo) - Special-case StreamingFunction in EagerThunkMetadata() constructor (#486) (@JamesWrigley) - Stop tracking the manifest files (#487) (@JamesWrigley) - Fix import of ScopedValues in Julia v1.11 (#489) (@JamesWrigley) - DArray: Implement efficient in-place matrix-matrix multiply (#492) (@Rabab53) - Add options docstrings to the docs (#493) (@JamesWrigley) - Doc/add Quickstart datadeps documentation (#495) (@johnmaxrin) - Streaming branch fixes (#496) (@JamesWrigley) - Remove more invalidations (#498) (@jpsamaroo) - build(deps): bump julia-actions/setup-julia from 1 to 2 (#499) (@dependabot[bot]) - Add num_processors helper (#500) (@jpsamaroo) - precompile: Don't let throwto hang the root task (#502) (@jpsamaroo)

Closed issues: - shard executes f in sequence, not in parallel (#473) - API Request: number of processors (including threads) (#474) - Feature Request: Support init argument for reduce functions. (#475) - Updating to Documenter.jl v1 (#482) - Cannot use Dagger with Julia 1.11 (alpha2) (#488) - Cannot use Dagger on Julia beta 1.11.0 (#501)

- Julia
Published by github-actions[bot] almost 2 years ago

Dagger - v0.18.8

Dagger v0.18.8

Diff since v0.18.7

Disabled CHUNK_CACHE eviction logic
Added view and cholesky/cholesky! operations to the DArray
Added helpers all_processors and compatible_processors
Fixed some precompile hangs seen on Mac M-series chips
Removed the DArray stage cache

Merged pull requests: - Sch: Disable CHUNK_CACHE evictions for now (#467) (@jpsamaroo) - DArray: Add Cholesky and view implementations (#469) (@jpsamaroo) - Add some processor utilities (#470) (@jpsamaroo) - precompile: Slightly lengthen pre-wait timeout (#471) (@jpsamaroo) - DArray: Remove the stage cache (#472) (@jpsamaroo)

- Julia
Published by github-actions[bot] about 2 years ago

Dagger - v0.18.7

Dagger v0.18.7

Diff since v0.18.6

Added spawn_datadeps for OpenMP-style (in/out) task dependencies
Eagerly free task results when task will no longer be used
Assorted bug fixes

Merged pull requests: - Add spawndatadeps for OMP-like task model (#454) (@jpsamaroo) - Retry jlsettasktid on failure (#462) (@jpsamaroo) - Eagerly free Thunk cached results when unneeded (#464) (@jpsamaroo) - Sch: Temporarily disable chunk cache (#465) (@jpsamaroo)

Closed issues: - Error: "jl_set_task_tid == 0" when calling length(::DTable) (#459)

- Julia
Published by github-actions[bot] about 2 years ago

Dagger - v0.18.6

Dagger v0.18.6

Diff since v0.18.5

Various bugfixes:

Removed some 1- and 2-letter exports that could conflict with user variables
Skipped accidental hashing of arbitrary task arguments in the scheduler
Fixed with_options not merging new options with previously-set options (yikes!)

Merged pull requests: - DaggerWebDash: Use gethostname and getipaddr (#451) (@jpsamaroo) - Don't export Bytes constants (#458) (@jpsamaroo) - reschedule_inputs: Don't hash non-Thunk inputs (#460) (@jpsamaroo) - tests: Test for previous option preservation (#461) (@jpsamaroo)

- Julia
Published by github-actions[bot] about 2 years ago

Dagger - v0.18.5

Dagger v0.18.5

Diff since v0.18.4

Fixed a variety of concurrency bugs

Merged pull requests: - Add precompilation via PrecompileTools (#446) (@jpsamaroo) - CompatHelper: add new compat entry for Statistics at version 1, (keep existing compat) (#447) (@github-actions[bot]) - processorrunner: Fix race in collect procstates (#452) (@jpsamaroo) - errormonitortracked: Use spin lock (#455) (@jpsamaroo) - OSProc: Add locking around OSPROCPROCESSOR_CACHE (#456) (@jpsamaroo)

Closed issues: - Failed to cleanup datastore after 0 seconds Forcibly evicting all entries (#445) - UNHANDLED TASK ERROR: ArgumentError: destination has fewer elements than required (#449) - DTables.groupby causes issues when multiple processes available (#450) - Occasional UndefRefError: access to undefined reference when fetching DTables (#453)

- Julia
Published by github-actions[bot] over 2 years ago

Dagger - v0.18.4

Dagger v0.18.4

Diff since v0.18.3

Merged pull requests: - options: Use with over deprecated scoped (#442) (@jpsamaroo) - Sch: Fix various locking and WeakChunk issues (#443) (@jpsamaroo)

Closed issues: - "Multiple concurrent writes to Dict detected!" with DTables.reduce (#437) - scoped deprecated (#441)

- Julia
Published by github-actions[bot] over 2 years ago

Dagger - v0.18.3

Dagger v0.18.3

Diff since v0.18.2

Merged pull requests: - Fix broken webpage link in README.md (#424) (@kylebeggs) - Update path splitting rule in Dagger.File / Dagger.tofile (#425) (@svilupp) - Fix error in documentation (#427) (@meggart) - Use ScopedValues instead of ContextVariablesX (#428) (@jpsamaroo) - build(deps): bump actions/checkout from 3 to 4 (#429) (@dependabot[bot]) - submission: Call init_eager on worker 1 (#432) (@jpsamaroo) - ThunkFailedException: Scrub out Thunk (#434) (@jpsamaroo) - submission: Fix variable shadowing bug (#436) (@jpsamaroo)

Closed issues: - type Nothing has no field thunk_dict with Distributed (#431) - Cannot serialize a Thunk when processing an error with Distributed (#430) - KeyError with selecting multiple columns (also with reduce and combine) with Distributed (#435)

- Julia
Published by github-actions[bot] over 2 years ago

Dagger - v0.18.2

Dagger v0.18.2

Diff since v0.18.1

Merged pull requests: - Requires-ify UI deps (#417) (@jpsamaroo) - Update Dagger README.md docs with a use case sceanario (#418) (@pszufe) - Add File and tofile helpers (#419) (@jpsamaroo)

- Julia
Published by github-actions[bot] over 2 years ago

Dagger - v0.18.1

Dagger v0.18.1

Diff since v0.18.0

Closed issues: - Outdated StatsBase version bound (#411)

Merged pull requests: - Add CompatHelper, UB StatsBase to 0.34 (#412) (@jpsamaroo) - TEST: Try implementing custom actions (#413) (@jpsamaroo) - disk_caching: Pass Dagger module in remotecall (#414) (@jpsamaroo) - build(deps): bump actions/checkout from 2 to 3 (#415) (@dependabot[bot]) - CI: Only check for custom cmds once (#416) (@jpsamaroo)

- Julia
Published by github-actions[bot] over 2 years ago

Dagger - v0.18.0

Dagger v0.18.0

Diff since v0.17.0

Merged pull requests: - Add DArray documentation (#336) (@mkitti) - enable_disk_caching function for handling MemPool storage setup (#361) (@krynju) - enable dependabot for GitHub actions (#382) (@ranocha) - DArray: Use eager API (#396) (@fda-tome) - Implement task queues for configurable task launch (#398) (@jpsamaroo) - Simplify and streamline docs, add Shard iteration (#399) (@jpsamaroo) - build(deps): bump actions/checkout from 2 to 3 (#400) (@dependabot[bot]) - DArray: Figure out correct eltype (#401) (@jpsamaroo) - DArray: Improve DiffEq support (#402) (@jpsamaroo) - Add logo to README (#403) (@jpsamaroo) - README: Add better logo (#404) (@jpsamaroo) - DArray: Operations always return DArrays, and add local partition support for MPI (#408) (@jpsamaroo) - at-mutable: Add remote execution like shard (#409) (@jpsamaroo) - Bump to 0.18.0, LB MemPool to 0.4.3 (#410) (@jpsamaroo)

- Julia
Published by github-actions[bot] over 2 years ago

Dagger - v0.17.0

Dagger v0.17.0

Diff since v0.16.3

Closed issues: - Determine work assignment and data movement based on runtime-collected metrics (#125) - Revitalize file I/O, aka swap-to-disk (#214) - DaggerWebDash.jl Instructions Broken (#376)

Merged pull requests: - Fix incorrect assertion in schedule! (#370) (@jpsamaroo) - checkpoint: Use at-invokelatest (#371) (@jpsamaroo) - at-spawn: Add support for broadcasting (#372) (@jpsamaroo) - Implement work stealing (#373) (@jpsamaroo) - Add ProcessorTypeScope, deprecate proclist and single (#374) (@jpsamaroo) - Update scheduler visualization docs (#377) (@jpsamaroo) - Updating README.md (#383) (@Madhupatel08) - Add keyword argument support (#394) (@jpsamaroo)

- Julia
Published by github-actions[bot] over 2 years ago

Dagger - v0.16.3

Dagger v0.16.3

Diff since v0.16.2

Closed issues: - DaggerWebDash using old libs (#365)

Merged pull requests: - signature: Don't capture input arguments (#363) (@jpsamaroo) - chunks: Allow weak Chunk references in Thunk args (#364) (@jpsamaroo) - Add DTables.jl CI run (#366) (@krynju) - DaggerWebDash: Add Mux 1.x to compat (#367) (@jpsamaroo)

- Julia
Published by github-actions[bot] over 3 years ago

Dagger - v0.16.2

Dagger v0.16.2

Diff since v0.16.1

Merged pull requests: - Split out logging as TimespanLogging.jl (#354) (@jpsamaroo) - DaggerWebDash: Improvements and generalization (#355) (@jpsamaroo) - Add missing check to walk_data (#359) (@jpsamaroo)

- Julia
Published by github-actions[bot] over 3 years ago

Dagger - v0.16.1

Dagger v0.16.1

Diff since v0.16.0

Merged pull requests: - Sch: Fix Some{Any} in CHUNK_CACHE (#353) (@jpsamaroo)

- Julia
Published by github-actions[bot] over 3 years ago

Dagger - v0.16.0

Dagger v0.16.0

Diff since v0.15.1

Merged pull requests: - Add memory and storage awareness (#289) (@jpsamaroo) - eager: Force kill scheduler on exit (#351) (@jpsamaroo) - WebDash: Add direct Tables dep, add TableStorage (#352) (@jpsamaroo)

- Julia
Published by github-actions[bot] over 3 years ago

Dagger - v0.15.1

Dagger v0.15.1

Diff since v0.15.0

Merged pull requests: - Remove poolset kwarg that doesn't exist anymore (#350) (@krynju)

- Julia
Published by github-actions[bot] over 3 years ago

Dagger - v0.15.0

Dagger v0.15.0

Diff since v0.14.4

Closed issues: - [feature request] @sync macro (#260)

Merged pull requests: - Switch to new TagBot (#343) (@giordano) - spawn: Disallow passing Thunk as argument (#345) (@jpsamaroo) - Minimize overhead of add_thunk! (#346) (@jpsamaroo) - at-spawn: Synchronize with at-sync (#347) (@jpsamaroo) - docs: Add at-sync synchronization section (#348) (@jpsamaroo) - Move DTable to separate repo (#349) (@krynju)

- Julia
Published by github-actions[bot] over 3 years ago

Dagger - v0.14.4

Dagger v0.14.4

Diff since v0.14.3

Closed issues: - dynamic_listener! not called for procs added after scheduler has started (#325) - Starting multiple Thunks effectively (#333) - Can't run README example (#338) - Execution slow for a simple sequence (#341)

Merged pull requests: - Sch: Start dynamic listener in initproc (#329) (@jpsamaroo) - Fix remove workers test (#332) (@DrChainsaw) - thunk,Sch: Reduce invalidations and TTFX (#334) (@jpsamaroo) - Sch,TTFX: Reduce more inference costs (#335) (@jpsamaroo) - Context: Add global context for lazy API (#337) (@jpsamaroo) - addthunk: Optimize domination check (#342) (@jpsamaroo)

- Julia
Published by github-actions[bot] almost 4 years ago

Dagger - v0.14.3

Dagger v0.14.3

Diff since v0.14.2

Merged pull requests: - Add with_options for propagating options (#328) (@jpsamaroo)

- Julia
Published by github-actions[bot] about 4 years ago

Dagger - v0.14.2

Dagger v0.14.2

Diff since v0.14.1

Closed issues: - Race in systemuuidfallback (#316) - DaggerWebDash not starting (#319) - [DTable] Loading .csv through the loader_function constructor takes allocates double the memory it's supposed to (#320)

Merged pull requests: - Don't clobber the scheduler (#310) (@jpsamaroo) - Make system UUID determination atomic (#317) (@jpsamaroo)

- Julia
Published by github-actions[bot] about 4 years ago

Dagger - v0.14.1

Dagger v0.14.1

Diff since v0.14.0

Closed issues: - Method ambiguity for simple Thunks (#314)

Merged pull requests: - Update LICENSE.md (#313) (@ViralBShah) - don't sort! parents when empty in order() (#315) (@mattwigway)

- Julia
Published by github-actions[bot] about 4 years ago

Dagger - v0.14.0

Dagger v0.14.0

Diff since v0.13.7

Closed issues: - Add MutableChunk for memory reuse (#186) - Data sharding and replication (#189) - Freeing unreferenced chunks used as input (#299) - DTable missing in Documentation API list (#302)

Merged pull requests: - [DTable] Leftjoin (#287) (@krynju) - at-spawn: Don't capture args in variable (#300) (@jpsamaroo) - Use juliaecosystem CI queue (#301) (@staticfloat) - docs: Add DTable functions/types to API docs (#303) (@jpsamaroo) - DTable Tables.jl interface adjustments (#304) (@krynju) - Add mutation helpers (#305) (@jpsamaroo) - docs: Reword and remove garbage (#307) (@jpsamaroo)