Releases | Open Source Science

https://github.com/cubed-dev/cubed - 0.22.0

What's Changed

Remove skipna=False from the Xarray examples by @tomwhite in https://github.com/cubed-dev/cubed/pull/706
Spark executor by @songhan89 in https://github.com/cubed-dev/cubed/pull/704
Update executor docs with new experimental executors by @tomwhite in https://github.com/cubed-dev/cubed/pull/705
Icechunk example by @tomwhite in https://github.com/cubed-dev/cubed/pull/708 and https://github.com/cubed-dev/cubed/pull/709
Remove newaxis from mapoverlap kwargs by @tomwhite in https://github.com/cubed-dev/cubed/pull/710
Make mypy ignore vendored libraries by @tomwhite in https://github.com/cubed-dev/cubed/pull/716
Fix doc build by @tomwhite in https://github.com/cubed-dev/cubed/pull/717
Drop Python 3.10 by @tomwhite in https://github.com/cubed-dev/cubed/pull/718
Add scan by @dcherian in https://github.com/cubed-dev/cubed/pull/531
Change list of output arrays in BlockwiseSpec to a map by @tomwhite in https://github.com/cubed-dev/cubed/pull/724
Simplify resume implementation by taking out of runtime by @tomwhite in https://github.com/cubed-dev/cubed/pull/725
Mem utilization tests fail on zarr-python v3 by @tomwhite in https://github.com/cubed-dev/cubed/pull/697
Minor improvements to rechunk_new by @tomwhite in https://github.com/cubed-dev/cubed/pull/732
Use generalblockwise for scan, rather than mapdirect by @tomwhite in https://github.com/cubed-dev/cubed/pull/726
Run array API tests on Zarr Python 2 and 3 by @tomwhite in https://github.com/cubed-dev/cubed/pull/733
Add MemoryVisualizationCallback by @tomwhite in https://github.com/cubed-dev/cubed/pull/734
Ruff by @tomwhite in https://github.com/cubed-dev/cubed/pull/735
Explicitly skip test_resume with tensorstore by @tomwhite in https://github.com/cubed-dev/cubed/pull/736
Allow use of obstore as a Zarr store by @tomwhite in https://github.com/cubed-dev/cubed/pull/723
Fix cardinality of outputdtypes in applygufunc by @tomwhite in https://github.com/cubed-dev/cubed/pull/738
Fix chunking where indexing an array returns an empty array by @tomwhite in https://github.com/cubed-dev/cubed/pull/739
Add a blocks property to Cubed arrays to allow block-level indexing by @tomwhite in https://github.com/cubed-dev/cubed/pull/737
Migrate to Icechunk 1.0 ForkSessions by @tomwhite in https://github.com/cubed-dev/cubed/pull/742
Use new rechunk implementation by default by @tomwhite in https://github.com/cubed-dev/cubed/pull/741
Remove zarray and zarray_maybe_lazy properties from Cubed Array by @tomwhite in https://github.com/cubed-dev/cubed/pull/743
Move index to indexing.py module by @tomwhite in https://github.com/cubed-dev/cubed/pull/744
Don't measure coverage for new executors by @tomwhite in https://github.com/cubed-dev/cubed/pull/745
Remove Spec from getting started doc by @tomwhite in https://github.com/cubed-dev/cubed/pull/746
Fix successors_unordered by @tomwhite in https://github.com/cubed-dev/cubed/pull/747
Exclude array API test that attempts in-place mutation by @tomwhite in https://github.com/cubed-dev/cubed/pull/754
Only run Zarr v2 CI tests on default local executors by @tomwhite in https://github.com/cubed-dev/cubed/pull/759
Update lithops aws image by @tomwhite in https://github.com/cubed-dev/cubed/pull/755
Fix indexing zero-sized dimension by @tomwhite in https://github.com/cubed-dev/cubed/pull/761
Run lithops CI on ubuntu-22.04 by @tomwhite in https://github.com/cubed-dev/cubed/pull/762

New Contributors

@songhan89 made their first contribution in https://github.com/cubed-dev/cubed/pull/704
@dcherian made their first contribution in https://github.com/cubed-dev/cubed/pull/531

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.21.0...0.22.0

- Python
Published by tomwhite 11 months ago

https://github.com/cubed-dev/cubed - 0.21.0

What's Changed

Icechunk store by @tomwhite in https://github.com/cubed-dev/cubed/pull/633
Move retry logic to lithops by @tomwhite in https://github.com/cubed-dev/cubed/pull/673
Coverage improvements by @tomwhite in https://github.com/cubed-dev/cubed/pull/674
Add scalar support to where by @tomwhite in https://github.com/cubed-dev/cubed/pull/679
Add PendingDeprecationWarning for map_direct by @tomwhite in https://github.com/cubed-dev/cubed/pull/682
Add a rechunking example by @tomwhite in https://github.com/cubed-dev/cubed/pull/681
Remove Lithops retries tests since this is now tested upstream by @tomwhite in https://github.com/cubed-dev/cubed/pull/683
Using default_dtypes instead of hard-coding dtypes by @alxmrs in https://github.com/cubed-dev/cubed/pull/666
Array API status update for draft 2024 spec by @tomwhite in https://github.com/cubed-dev/cubed/pull/684
Refactor asyncio executors to share common code by @tomwhite in https://github.com/cubed-dev/cubed/pull/685
Add scalar support to binary element-wise ops by @tomwhite in https://github.com/cubed-dev/cubed/pull/686
Fix icechunk tests to opt-in to pickling of Session by @tomwhite in https://github.com/cubed-dev/cubed/pull/688
Move non-standardized functions to separate package by @tomwhite in https://github.com/cubed-dev/cubed/pull/690
Add Ray executor by @tomwhite in https://github.com/cubed-dev/cubed/pull/687
Allow nan functions to work with xarray by @tomwhite in https://github.com/cubed-dev/cubed/pull/670
Make callbacks context managers by @tomwhite in https://github.com/cubed-dev/cubed/pull/691
Adding pre-commit (for linting) to test dependencies by @alxmrs in https://github.com/cubed-dev/cubed/pull/671
Support float32 in random number generation by @tomwhite in https://github.com/cubed-dev/cubed/pull/693
Fix for Zarr 3.0.3 change to OrthogonalIndexer internals from Zarr #2784 by @tomwhite in https://github.com/cubed-dev/cubed/pull/696
Modal 1.0 changes by @tomwhite in https://github.com/cubed-dev/cubed/pull/695
Show nchunks on visualization by @tomwhite in https://github.com/cubed-dev/cubed/pull/698
Always set region for Modal executor by @tomwhite in https://github.com/cubed-dev/cubed/pull/699
Improve memory model by explicitly modelling buffer copies by @tomwhite in https://github.com/cubed-dev/cubed/pull/701
Multistage rechunking by @tomwhite in https://github.com/cubed-dev/cubed/pull/700
Remove duplicate code from new rechunk implementation by @tomwhite in https://github.com/cubed-dev/cubed/pull/702
Use extra buffer copies for cloud stores like S3 by @tomwhite in https://github.com/cubed-dev/cubed/pull/703

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.20.0...0.21.0

- Python
Published by tomwhite about 1 year ago

https://github.com/cubed-dev/cubed - 0.20.0

What's Changed

Include task result in callbacks by @tomwhite in https://github.com/cubed-dev/cubed/pull/632
Pin dask and distributed to < 2024.12.0 by @tomwhite in https://github.com/cubed-dev/cubed/pull/639
Fix Modal deprecation (show progress) and pip package by @tomwhite in https://github.com/cubed-dev/cubed/pull/634
Remove test 'extras' and run lithops tests separately by @tomwhite in https://github.com/cubed-dev/cubed/pull/640
Array API tests fixes by @tomwhite in https://github.com/cubed-dev/cubed/pull/636
New tagline by @tomwhite in https://github.com/cubed-dev/cubed/pull/641
Add tile by @tomwhite in https://github.com/cubed-dev/cubed/pull/644
Add searchsorted by @tomwhite in https://github.com/cubed-dev/cubed/pull/647
Declare Array API 2023.12 support by @tomwhite in https://github.com/cubed-dev/cubed/pull/651
Change chunk_shape back to chunks for Zarr v3 by @tomwhite in https://github.com/cubed-dev/cubed/pull/656
Document use of cubed.config by @tomwhite in https://github.com/cubed-dev/cubed/pull/654
Make async executors work in Jupyter notebooks by @tomwhite in https://github.com/cubed-dev/cubed/pull/661
Add Zarr and Xarray examples to docs by @tomwhite in https://github.com/cubed-dev/cubed/pull/655
Fix tooltips for SVGs generated by visualize() in Jupyter notebooks by @tomwhite in https://github.com/cubed-dev/cubed/pull/663
Add note to contributing guide for developers to have graphviz installed by @alxmrs in https://github.com/cubed-dev/cubed/pull/665
Incorrect dtypes in map_selection by @tomwhite in https://github.com/cubed-dev/cubed/pull/669

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.19.0...0.20.0

- Python
Published by tomwhite over 1 year ago

https://github.com/cubed-dev/cubed - 0.19.0

What's Changed

Support block_id for general_blockwise functions by @tomwhite in https://github.com/cubed-dev/cubed/pull/593
Run tests on Python 3.13 by @tomwhite in https://github.com/cubed-dev/cubed/pull/503
Fix dtypes for QR by @tomwhite in https://github.com/cubed-dev/cubed/pull/594
Change usages of map_direct in QR to general_blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/597
Implement var and std using a numerically stable parallel algorithm by @tomwhite in https://github.com/cubed-dev/cubed/pull/596
Add map_selection primitive to replace map_direct by @tomwhite in https://github.com/cubed-dev/cubed/pull/595
Use unstack from array-api-compat by @tomwhite in https://github.com/cubed-dev/cubed/pull/602
Increase allowed_mem default to 2GB by @tomwhite in https://github.com/cubed-dev/cubed/pull/599
Implement structured arrays as Zarr group of arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/603
Implement int array indexing using map_selection by @tomwhite in https://github.com/cubed-dev/cubed/pull/604
Implement mergechunks using mapselection by @tomwhite in https://github.com/cubed-dev/cubed/pull/605
Free memory earlier by changing function scope by @tomwhite in https://github.com/cubed-dev/cubed/pull/606
Fix groupby tests running on tensorstore by @tomwhite in https://github.com/cubed-dev/cubed/pull/608
Implement concat using general_blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/607
Speed up test collection by @tomwhite in https://github.com/cubed-dev/cubed/pull/609
Update GH actions to macos-13 test runner and upload-artifact@v4 by @tomwhite in https://github.com/cubed-dev/cubed/pull/611
Implement repeat by @tomwhite in https://github.com/cubed-dev/cubed/pull/610
Map nested improvements by @tomwhite in https://github.com/cubed-dev/cubed/pull/613
SVD by @tomwhite in https://github.com/cubed-dev/cubed/pull/598
Unify chunks for concat, and check preconditions by @tomwhite in https://github.com/cubed-dev/cubed/pull/616
Defer to merge_chunks in special cases of rechunk by @tomwhite in https://github.com/cubed-dev/cubed/pull/612
Document map_selection core operation by @tomwhite in https://github.com/cubed-dev/cubed/pull/617
Change default max_total_num_input_blocks to 10 by @tomwhite in https://github.com/cubed-dev/cubed/pull/615
Update slides - November 2024 by @tomwhite in https://github.com/cubed-dev/cubed/pull/620
Change use_backups default to True only on cloud stores by @tomwhite in https://github.com/cubed-dev/cubed/pull/619
Various minor doc improvements by @tomwhite in https://github.com/cubed-dev/cubed/pull/625
Make threads the default executor by @tomwhite in https://github.com/cubed-dev/cubed/pull/621
Rewrite Cubed intro docs by @tomwhite in https://github.com/cubed-dev/cubed/pull/622
More doc improvements: simplify getting started page by @tomwhite in https://github.com/cubed-dev/cubed/pull/629
Add examples to docs by @tomwhite in https://github.com/cubed-dev/cubed/pull/630

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.18.0...0.19.0

- Python
Published by tomwhite over 1 year ago

https://github.com/cubed-dev/cubed - 0.18.0

What's Changed

Multiple outputs by @tomwhite in https://github.com/cubed-dev/cubed/pull/419
Support use of multiprocessing start methods other than "spawn" (e.g. "dragon") by @applio in https://github.com/cubed-dev/cubed/pull/554
Tensorstore compression fix by @tomwhite in https://github.com/cubed-dev/cubed/pull/574
Implement unstack using multiple outputs by @tomwhite in https://github.com/cubed-dev/cubed/pull/575
Handle more memray allocation types by @tomwhite in https://github.com/cubed-dev/cubed/pull/578
Drop Python 3.9 by @tomwhite in https://github.com/cubed-dev/cubed/pull/576
QR decomposition by @tomwhite in https://github.com/cubed-dev/cubed/pull/577
Expose the linalg namespace and include in status page by @tomwhite in https://github.com/cubed-dev/cubed/pull/581
Fix outer result dtype by @tomwhite in https://github.com/cubed-dev/cubed/pull/582
Add new 2023.12 elemwise functions by @tomwhite in https://github.com/cubed-dev/cubed/pull/583
Bring vecdot implementation in line with the one in array-api-compat by @tomwhite in https://github.com/cubed-dev/cubed/pull/402
Zarr v3 now requires at least Python 3.11 by @tomwhite in https://github.com/cubed-dev/cubed/pull/585
Debug failing straggler test by @tomwhite in https://github.com/cubed-dev/cubed/pull/580
Set object codec for object arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/573
Fix for flaky straggler test by @tomwhite in https://github.com/cubed-dev/cubed/pull/587
Use general_blockwise in case of chunk-aligned selections in index by @tomwhite in https://github.com/cubed-dev/cubed/pull/586
Remove deprecated code by @tomwhite in https://github.com/cubed-dev/cubed/pull/588
Documentation small fixes by @tomwhite in https://github.com/cubed-dev/cubed/pull/591
Add page on optimization to user guide by @tomwhite in https://github.com/cubed-dev/cubed/pull/397
Remove old reduction implementation by @tomwhite in https://github.com/cubed-dev/cubed/pull/589
Add array API inspection utilities by @tomwhite in https://github.com/cubed-dev/cubed/pull/592

New Contributors

@applio made their first contribution in https://github.com/cubed-dev/cubed/pull/554

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.17.0...0.18.0

- Python
Published by tomwhite over 1 year ago

https://github.com/cubed-dev/cubed - 0.17.0

What's Changed

Skip fewer array API tests by @tomwhite in https://github.com/cubed-dev/cubed/pull/547
Rechunk where dict has missing axes by @tomwhite in https://github.com/cubed-dev/cubed/pull/546
Adds logging [INFO] to TimelineVisualizationCallback and HistoryCallback by @norlandrhagen in https://github.com/cubed-dev/cubed/pull/548
More lenient dtype support by @tomwhite in https://github.com/cubed-dev/cubed/pull/550
Cast inputs to Cubed arrays in apply_ufunc by @tomwhite in https://github.com/cubed-dev/cubed/pull/551
Fix bug where newaxis with full slices doesn't add new axes by @tomwhite in https://github.com/cubed-dev/cubed/pull/559
Memray integration by @tomwhite in https://github.com/cubed-dev/cubed/pull/558
Add chunkmem to VirtualFullArray, which was missed in #447 by @tomwhite in https://github.com/cubed-dev/cubed/pull/560
Fix Array API Tests by @tomwhite in https://github.com/cubed-dev/cubed/pull/562
Memray callback by @tomwhite in https://github.com/cubed-dev/cubed/pull/561
Introduce FinalizedPlan by @tomwhite in https://github.com/cubed-dev/cubed/pull/563
Materialize arrays optimizer bugfix by @tomwhite in https://github.com/cubed-dev/cubed/pull/564
Align tqdm progress bars by @tomwhite in https://github.com/cubed-dev/cubed/pull/567
Coerce args to map_blocks to arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/566
Fail during planning if mapblocks dropaxis is for a chunked dimension by @tomwhite in https://github.com/cubed-dev/cubed/pull/569
Fix Rich progress bar flickering by limiting refreshes by @tomwhite in https://github.com/cubed-dev/cubed/pull/571
Allow Zarr compression to be set for intermediate files by @tomwhite in https://github.com/cubed-dev/cubed/pull/572

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.16.0...0.17.0

- Python
Published by tomwhite over 1 year ago

https://github.com/cubed-dev/cubed - 0.16.0

What's Changed

Improve documentation around Array API support by @tomwhite in https://github.com/cubed-dev/cubed/pull/477
Speed up tests by @JosepSampe in https://github.com/cubed-dev/cubed/pull/478
Add normalize_shape since it is not in Zarr's public API by @tomwhite in https://github.com/cubed-dev/cubed/pull/479
Implement virtual array indexing using ndindex by @tomwhite in https://github.com/cubed-dev/cubed/pull/441
Ndindex indexing by @tomwhite in https://github.com/cubed-dev/cubed/pull/481
Pluggable backend storage by @tomwhite in https://github.com/cubed-dev/cubed/pull/480
Ensure integer arrays used for indexing are NumPy arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/485
Indicate that we support both 2021.12 and 2022.12 versions of the Array API by @tomwhite in https://github.com/cubed-dev/cubed/pull/487
TensorStore storage by @tomwhite in https://github.com/cubed-dev/cubed/pull/483
Update slides - June 2024 by @tomwhite in https://github.com/cubed-dev/cubed/pull/489
Fail with NotImplementedError if backend Zarr is missing nchunks_initialized by @tomwhite in https://github.com/cubed-dev/cubed/pull/491
Zarr v3 by @tomwhite in https://github.com/cubed-dev/cubed/pull/484
Remove print for local executors by @tomwhite in https://github.com/cubed-dev/cubed/pull/506
document what needs to be installed to run climatology anomaly example by @rbavery in https://github.com/cubed-dev/cubed/pull/501
Document how to run examples on local machine using processes by @tomwhite in https://github.com/cubed-dev/cubed/pull/507
Check allowed_mem does not exceed total on machine for processes executor by @tomwhite in https://github.com/cubed-dev/cubed/pull/517
Warn if peak mem exceeds allowed_mem by @tomwhite in https://github.com/cubed-dev/cubed/pull/516
Remove max_tasks_per_child=1 limitation from processes executor by @tomwhite in https://github.com/cubed-dev/cubed/pull/515
Zarr Python v3 updates by @tomwhite in https://github.com/cubed-dev/cubed/pull/523
Modal progress fix by @tomwhite in https://github.com/cubed-dev/cubed/pull/522
Delete local context directories on exit by @tomwhite in https://github.com/cubed-dev/cubed/pull/524
Update demo notebooks by @tomwhite in https://github.com/cubed-dev/cubed/pull/525
Update CUBED_CONFIG path in example README.md by @thodson-usgs in https://github.com/cubed-dev/cubed/pull/530
Implement flip by @tomwhite in https://github.com/cubed-dev/cubed/pull/528
Add better test for local paths in delete_on_exit by @tomwhite in https://github.com/cubed-dev/cubed/pull/532
Deprecate cubed.extensions and move to cubed.diagnostics by @tomwhite in https://github.com/cubed-dev/cubed/pull/533
Turning on JAX tests for all PRs. by @alxmrs in https://github.com/cubed-dev/cubed/pull/537
Improve set-up instructions for examples by @tomwhite in https://github.com/cubed-dev/cubed/pull/534
Configuration documentation by @tomwhite in https://github.com/cubed-dev/cubed/pull/535
Adding compile_function as execute option. by @alxmrs in https://github.com/cubed-dev/cubed/pull/536
Diagnostics documentation by @tomwhite in https://github.com/cubed-dev/cubed/pull/540

New Contributors

@JosepSampe made their first contribution in https://github.com/cubed-dev/cubed/pull/478
@rbavery made their first contribution in https://github.com/cubed-dev/cubed/pull/501
@thodson-usgs made their first contribution in https://github.com/cubed-dev/cubed/pull/530
@alxmrs made their first contribution in https://github.com/cubed-dev/cubed/pull/537

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.15.0...0.16.0

- Python
Published by tomwhite almost 2 years ago

https://github.com/cubed-dev/cubed - 0.15.0

What's Changed

Test on Python 3.12 by @tomwhite in https://github.com/cubed-dev/cubed/pull/453
Test NumPy 2 by @tomwhite in https://github.com/cubed-dev/cubed/pull/454
Expose Lithops wait_dur_sec to speed up tests by @tomwhite in https://github.com/cubed-dev/cubed/pull/456
Simplify executor naming by @tomwhite in https://github.com/cubed-dev/cubed/pull/457
Treat warnings as errors when running tests by @tomwhite in https://github.com/cubed-dev/cubed/pull/458
Re-interpret 'fusable' to mean with predecessor operations by @tomwhite in https://github.com/cubed-dev/cubed/pull/460
Don't include hidden arrays when counting max_total_source_arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/463
Limited implementation of map_overlap by @tomwhite in https://github.com/cubed-dev/cubed/pull/462
Limited implementation of pad by @tomwhite in https://github.com/cubed-dev/cubed/pull/461
Implement roll by @tomwhite in https://github.com/cubed-dev/cubed/pull/465
Fix map_overlap in case of single chunk in overlap dimension by @tomwhite in https://github.com/cubed-dev/cubed/pull/466
Fix case of Lithops 3.4.0 hanging by @tomwhite in https://github.com/cubed-dev/cubed/pull/471
Fix unify_chunks to return regular chunks in all cases by @tomwhite in https://github.com/cubed-dev/cubed/pull/470
Improve test coverage by @tomwhite in https://github.com/cubed-dev/cubed/pull/472
Change development status to beta by @tomwhite in https://github.com/cubed-dev/cubed/pull/474
Top level array namespace by @tomwhite in https://github.com/cubed-dev/cubed/pull/473
Add notebooks for Pangeo examples by @tomwhite in https://github.com/cubed-dev/cubed/pull/476

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.14.3...0.15.0

- Python
Published by tomwhite almost 2 years ago

https://github.com/cubed-dev/cubed - 0.14.3

What's Changed

Take broadcast trick into account for array chunk memory by @tomwhite in https://github.com/cubed-dev/cubed/pull/447
Fix Documentation link by @norlandrhagen in https://github.com/cubed-dev/cubed/pull/451
Include j2 templates in the distribution by @tomwhite in https://github.com/cubed-dev/cubed/pull/450
Add links to blog posts by @TomNicholas in https://github.com/cubed-dev/cubed/pull/452
Add a groupby_blockwise function for use in Flox by @tomwhite in https://github.com/cubed-dev/cubed/pull/448

New Contributors

@norlandrhagen made their first contribution in https://github.com/cubed-dev/cubed/pull/451

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.14.2...0.14.3

- Python
Published by tomwhite about 2 years ago

https://github.com/cubed-dev/cubed - 0.14.2

What's Changed

Allow networkx 3.3 now that pydot issue has been fixed by @tomwhite in https://github.com/cubed-dev/cubed/pull/445
Change drop_axis to default to None rather than an empty list in map_blocks by @tomwhite in https://github.com/cubed-dev/cubed/pull/446
Add a groupby_reduction function for use in Flox by @tomwhite in https://github.com/cubed-dev/cubed/pull/442

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.14.1...0.14.2

- Python
Published by tomwhite about 2 years ago

https://github.com/cubed-dev/cubed - 0.14.1

What's Changed

Providing filesystem credentials through storage_options by @balanz24 in https://github.com/cubed-dev/cubed/pull/436
Only use oindex for advanced indexing (int arrays) by @tomwhite in https://github.com/cubed-dev/cubed/pull/440
Deprecate misspelled aggegrate_func in favour of aggregate_func by @tomwhite in https://github.com/cubed-dev/cubed/pull/444

New Contributors

@balanz24 made their first contribution in https://github.com/cubed-dev/cubed/pull/436

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.14.0...0.14.1

- Python
Published by tomwhite about 2 years ago

https://github.com/cubed-dev/cubed - 0.14.0

What's Changed

Handle zero-length slices in index function by @tomwhite in https://github.com/cubed-dev/cubed/pull/401
Run unit tests on Python 3.11 on Linux by @tomwhite in https://github.com/cubed-dev/cubed/pull/405
Update examples to use Python 3.11 by @tomwhite in https://github.com/cubed-dev/cubed/pull/406
Remove unused lazy_full function (replaced by virtual_full) by @tomwhite in https://github.com/cubed-dev/cubed/pull/409
Delegate dtype functions to backend array API by @tomwhite in https://github.com/cubed-dev/cubed/pull/410
Add a 'processes' executor that runs using local processes by @tomwhite in https://github.com/cubed-dev/cubed/pull/411
Mark stack operation as not fusable by @tomwhite in https://github.com/cubed-dev/cubed/pull/415
Track source array names in PrimitiveOperation by @tomwhite in https://github.com/cubed-dev/cubed/pull/416
Fix Modal deprecations by @tomwhite in https://github.com/cubed-dev/cubed/pull/417
Make the new multiple-inputs optimization function the default by @tomwhite in https://github.com/cubed-dev/cubed/pull/412
Add cloudpickle dependency following #411 ('processes' executor) by @tomwhite in https://github.com/cubed-dev/cubed/pull/421
Documentation corrections following optimization improvements by @tomwhite in https://github.com/cubed-dev/cubed/pull/423
Add name property to all executors by @tomwhite in https://github.com/cubed-dev/cubed/pull/424
Build Lithops images for every push to main by @tomwhite in https://github.com/cubed-dev/cubed/pull/422
Make Lithops example names consistent with benchmarks and other executors by @tomwhite in https://github.com/cubed-dev/cubed/pull/425
Delete existing Lithops image before rebuilding by @tomwhite in https://github.com/cubed-dev/cubed/pull/427
Rename block_function to key_function in blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/426
Use executor name rather than class name by @tomwhite in https://github.com/cubed-dev/cubed/pull/430
Remove unused sync version of Modal executor by @tomwhite in https://github.com/cubed-dev/cubed/pull/431
Change total_nbytes to total_nbytes_written by @tomwhite in https://github.com/cubed-dev/cubed/pull/433
Fix deprecation warning on logger.warn by @tomwhite in https://github.com/cubed-dev/cubed/pull/434

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.13.0...0.14.0

- Python
Published by tomwhite about 2 years ago

https://github.com/cubed-dev/cubed - 0.13.0

What's Changed

Introduce finalize_dag method in Plan that caches DAG optimization by @tomwhite in https://github.com/cubed-dev/cubed/pull/340
Add VirtualInMemoryArray that keeps small arrays in memory by @tomwhite in https://github.com/cubed-dev/cubed/pull/336
Fix Modal test following #336 by @tomwhite in https://github.com/cubed-dev/cubed/pull/341
Change the DAG to have separate nodes for operations and arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/337
Allow more control over optimizations applied by @tomwhite in https://github.com/cubed-dev/cubed/pull/342
Handle the case where an array is created by calling map_blocks with no input arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/343
Add a test for visualizing the quad means computation by @tomwhite in https://github.com/cubed-dev/cubed/pull/345
Show nbytes for each array and total on visualization by @tomwhite in https://github.com/cubed-dev/cubed/pull/344
Fuse multiple inputs by @tomwhite in https://github.com/cubed-dev/cubed/pull/346
Upgrade to sphinx-notes/pages v3 by @tomwhite in https://github.com/cubed-dev/cubed/pull/347
Minor improvements to DAG visualization by @tomwhite in https://github.com/cubed-dev/cubed/pull/348
Remove deprecated code by @tomwhite in https://github.com/cubed-dev/cubed/pull/349
Add generated doc for primitive and core ops to design page by @tomwhite in https://github.com/cubed-dev/cubed/pull/351
Support arbitrary block index functions in blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/350
Refactor pipeline to reduce coupling between runtime and primitive layers by @tomwhite in https://github.com/cubed-dev/cubed/pull/352
Implement stack using general_blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/353
Implement reshape_chunks using general_blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/354
More control over operation fusion by @tomwhite in https://github.com/cubed-dev/cubed/pull/355
Minor memory fixes by @tomwhite in https://github.com/cubed-dev/cubed/pull/358
Use Dask's 'broadcast trick' to save memory for single-valued arrays by @tomwhite in https://github.com/cubed-dev/cubed/pull/359
Add a show_hidden option to visualize by @tomwhite in https://github.com/cubed-dev/cubed/pull/360
Reinstate slow tests by @tomwhite in https://github.com/cubed-dev/cubed/pull/361
Add function_nargs to BlockwiseSpec by @tomwhite in https://github.com/cubed-dev/cubed/pull/363
Add missing api doc by @tomwhite in https://github.com/cubed-dev/cubed/pull/364
Implement partial_reduce and tree_reduce using generalized blockwise by @tomwhite in https://github.com/cubed-dev/cubed/pull/365
Quad means validation test by @tomwhite in https://github.com/cubed-dev/cubed/pull/367
Take allowed_mem into account when fusing primitive operations by @tomwhite in https://github.com/cubed-dev/cubed/pull/366
Fuse operations with different numbers of tasks by @tomwhite in https://github.com/cubed-dev/cubed/pull/368
Show number of tasks prominently on graph visualization by @tomwhite in https://github.com/cubed-dev/cubed/pull/371
Introduce event objects for all callback methods by @tomwhite in https://github.com/cubed-dev/cubed/pull/373
Minor Modal improvements by @tomwhite in https://github.com/cubed-dev/cubed/pull/375
Run tests using JAX as the backend array API (on CPU) by @tomwhite in https://github.com/cubed-dev/cubed/pull/374
Introduce max_total_num_input_blocks as a heuristic to control fusion by @tomwhite in https://github.com/cubed-dev/cubed/pull/376
Rename array_name to name in runtime following #337 by @tomwhite in https://github.com/cubed-dev/cubed/pull/377
Enable backup tasks by default for all executors that support them by @tomwhite in https://github.com/cubed-dev/cubed/pull/378
Remove unused array_names parameter in executors by @tomwhite in https://github.com/cubed-dev/cubed/pull/380
Introduce compute_id for each computation by @tomwhite in https://github.com/cubed-dev/cubed/pull/382
Add a Rich progress bar by @tomwhite in https://github.com/cubed-dev/cubed/pull/383
Change timeline visualization default format to SVG by @tomwhite in https://github.com/cubed-dev/cubed/pull/384
Preserve laziness in fuse_multiple to avoid materializing array blocks in _partial_reduce by @tomwhite in https://github.com/cubed-dev/cubed/pull/386
Fix Modal unit tests following change in #382 by @tomwhite in https://github.com/cubed-dev/cubed/pull/387
Add debug log statements to optimizer to see why nodes are fused by @tomwhite in https://github.com/cubed-dev/cubed/pull/388
Allow optional path in from_zarr and to_zarr by @tomwhite in https://github.com/cubed-dev/cubed/pull/391
Remove CUBEDMODALFORCE_BUILD env var … by @tomwhite in https://github.com/cubed-dev/cubed/pull/393
Add introductory slides to docs by @tomwhite in https://github.com/cubed-dev/cubed/pull/394
Use donfig to control default spec by @tomwhite in https://github.com/cubed-dev/cubed/pull/392
small changes to coiled/aws examples by @rsignell in https://github.com/cubed-dev/cubed/pull/369
Skip vecdot test due to zero-size array failures by @tomwhite in https://github.com/cubed-dev/cubed/pull/399
Show both operation ID and name in visualization by @tomwhite in https://github.com/cubed-dev/cubed/pull/396
Add support for minimum_workers in Coiled by @tomwhite in https://github.com/cubed-dev/cubed/pull/398
Create a single set of example scripts that can run on any executor by @tomwhite in https://github.com/cubed-dev/cubed/pull/395

New Contributors

@rsignell made their first contribution in https://github.com/cubed-dev/cubed/pull/369

Full Changelog: https://github.com/cubed-dev/cubed/compare/0.12.0...0.13.0

- Python
Published by tomwhite over 2 years ago

https://github.com/cubed-dev/cubed - 0.12.0

What's Changed

Improve separation between layers by @tomwhite in https://github.com/tomwhite/cubed/pull/314
Fix Modal deprecation of f.call(...) by @tomwhite in https://github.com/tomwhite/cubed/pull/320
Disable the Hypthosis deadline when running Array API tests. by @tomwhite in https://github.com/tomwhite/cubed/pull/325
Allow dtype to be passed to blockwise and reduction functions by @tomwhite in https://github.com/tomwhite/cubed/pull/321
Array API backend by @tomwhite in https://github.com/tomwhite/cubed/pull/317
Update examples to use Python 3.9 by @tomwhite in https://github.com/tomwhite/cubed/pull/330
Re-write arange to use map_direct, which avoids writing a Zarr array by @tomwhite in https://github.com/tomwhite/cubed/pull/332

Full Changelog: https://github.com/tomwhite/cubed/compare/0.11.0...0.12.0

- Python
Published by tomwhite over 2 years ago

https://github.com/cubed-dev/cubed - 0.11.0

What's Changed

Introduce VirtualEmptyArray and hide virtual arrays in the DAG visualization by @tomwhite in https://github.com/tomwhite/cubed/pull/290
Introduce RetryingFunctionExecutor by @tomwhite in https://github.com/tomwhite/cubed/pull/292
Add support for batching inputs to avoid overwhelming the backend service by @tomwhite in https://github.com/tomwhite/cubed/pull/291
Support version 2022.12 of the Array API by @tomwhite in https://github.com/tomwhite/cubed/pull/293
Fix array __abs__ method for complex inputs by @tomwhite in https://github.com/tomwhite/cubed/pull/294
Be explicit when opening Zarr arrays by @tomwhite in https://github.com/tomwhite/cubed/pull/296
Fix bug when running with Kerchunk by @tomwhite in https://github.com/tomwhite/cubed/pull/297
Remove limited implementation of __setitem__ from Array by @tomwhite in https://github.com/tomwhite/cubed/pull/302
Introduce VirtualFullArray by @tomwhite in https://github.com/tomwhite/cubed/pull/303
[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/tomwhite/cubed/pull/305
Fix case where shape is an int for empty by @tomwhite in https://github.com/tomwhite/cubed/pull/306
Update Coiled example by @tomwhite in https://github.com/tomwhite/cubed/pull/309
Add nansum and nanmean functions by @tomwhite in https://github.com/tomwhite/cubed/pull/308
Add moveaxis by @tomwhite in https://github.com/tomwhite/cubed/pull/313
Update minimum Python version to 3.9 by @tomwhite in https://github.com/tomwhite/cubed/pull/298

Full Changelog: https://github.com/tomwhite/cubed/compare/0.10.0...0.11.0

- Python
Published by tomwhite over 2 years ago

https://github.com/cubed-dev/cubed - 0.10.0

What's Changed

Scaling docs by @TomNicholas in https://github.com/tomwhite/cubed/pull/252
Remove PipelineExecutor classes by @tomwhite in https://github.com/tomwhite/cubed/pull/253
Surface rechunk intermediates in DAG by @tomwhite in https://github.com/tomwhite/cubed/pull/257
Add type hints for runtime by @tomwhite in https://github.com/tomwhite/cubed/pull/258
Fix modal import by @tomwhite in https://github.com/tomwhite/cubed/pull/261
Add compute_arrays_in_parallel option to Modal and Python async executors by @tomwhite in https://github.com/tomwhite/cubed/pull/263
Use new merge_chunks function in reduction by @tomwhite in https://github.com/tomwhite/cubed/pull/256
Remove vendored rechunker API, since we only use the algorithm by @tomwhite in https://github.com/tomwhite/cubed/pull/264
Add unit tests for virtual_offsets function by @tomwhite in https://github.com/tomwhite/cubed/pull/265
[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/tomwhite/cubed/pull/266
Lithops compute arrays in parallel by @tomwhite in https://github.com/tomwhite/cubed/pull/259
Remove duplicate copy of projected_mem and num_tasks from DAG by @tomwhite in https://github.com/tomwhite/cubed/pull/271
Allow step size > 1 by @tomwhite in https://github.com/tomwhite/cubed/pull/270
Fix reserved_mem in create_lazy_zarr_arrays by @tomwhite in https://github.com/tomwhite/cubed/pull/273
Dask tests by @tomwhite in https://github.com/tomwhite/cubed/pull/269
Coiled Functions executor by @TomNicholas in https://github.com/tomwhite/cubed/pull/260
Ensure runtime memory is not less than allowed by @tomwhite in https://github.com/tomwhite/cubed/pull/274
In-process Python executor should not retry by @tomwhite in https://github.com/tomwhite/cubed/pull/275
Refactor pipeline mappable by @tomwhite in https://github.com/tomwhite/cubed/pull/272
Generalise async executor code by @tomwhite in https://github.com/tomwhite/cubed/pull/278
Dask distributed async executor by @tomwhite in https://github.com/tomwhite/cubed/pull/279
[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/tomwhite/cubed/pull/283
Dask checkruntimememory by @tomwhite in https://github.com/tomwhite/cubed/pull/281
Fixes for converting strings to bytes by @TomNicholas in https://github.com/tomwhite/cubed/pull/289

Full Changelog: https://github.com/tomwhite/cubed/compare/0.9.0...0.10.0

- Python
Published by tomwhite almost 3 years ago

https://github.com/cubed-dev/cubed - 0.9.0

What's Changed

Fix Modal deprecated API by @tomwhite in https://github.com/tomwhite/cubed/pull/214
Modal GCP by @tomwhite in https://github.com/tomwhite/cubed/pull/215
Add html array repr by @TomNicholas in https://github.com/tomwhite/cubed/pull/216
Add to pre-commit.ci by @tomwhite in https://github.com/tomwhite/cubed/pull/218
Skip brew update when setting up GraphViz on mac CI to improve run time by @tomwhite in https://github.com/tomwhite/cubed/pull/227
Set 'reraise' to True for retries by @tomwhite in https://github.com/tomwhite/cubed/pull/225
Run mypy in CI by @tomwhite in https://github.com/tomwhite/cubed/pull/229
Print irregular chunking pattern when error raised by @TomNicholas in https://github.com/tomwhite/cubed/pull/231
[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/tomwhite/cubed/pull/232
Allow rechunk to take a dict for chunks by @tomwhite in https://github.com/tomwhite/cubed/pull/226
Improve Modal backups implementation by @tomwhite in https://github.com/tomwhite/cubed/pull/238
Test that plan generation scales to very large workloads by @tomwhite in https://github.com/tomwhite/cubed/pull/240
Don't materialize input iterables to map_unordered by @tomwhite in https://github.com/tomwhite/cubed/pull/242
Lithops retries and backups by @tomwhite in https://github.com/tomwhite/cubed/pull/217
Add type hints for primitive functions by @tomwhite in https://github.com/tomwhite/cubed/pull/236
Add links to Xarray integration and blog post to docs by @tomwhite in https://github.com/tomwhite/cubed/pull/243
Account for reserved_mem when creating arrays by @tomwhite in https://github.com/tomwhite/cubed/pull/245
Use np.unravel_index rather than iterating over np.ndindex by @tomwhite in https://github.com/tomwhite/cubed/pull/246
Add matmul example that validates result by @tomwhite in https://github.com/tomwhite/cubed/pull/244
Document strong consistency by @tomwhite in https://github.com/tomwhite/cubed/pull/250
Re-enable rechunker's consolidate_reads by @tomwhite in https://github.com/tomwhite/cubed/pull/249
Introduce VirtualOffsetsArray that is never materialized by @tomwhite in https://github.com/tomwhite/cubed/pull/248
Avoid divide by zero from projected mem by @tomwhite in https://github.com/tomwhite/cubed/pull/251

New Contributors

@pre-commit-ci made their first contribution in https://github.com/tomwhite/cubed/pull/232

Full Changelog: https://github.com/tomwhite/cubed/compare/0.8.0...0.9.0

- Python
Published by tomwhite almost 3 years ago

https://github.com/cubed-dev/cubed - 0.8.0

What's Changed

Lithops gcf example by @TomNicholas in https://github.com/tomwhite/cubed/pull/183
Name Modal Stub objects to avoid warning by @tomwhite in https://github.com/tomwhite/cubed/pull/189
Convert str to bytes by @TomNicholas in https://github.com/tomwhite/cubed/pull/188
Reinstate Lithops AWS examples that were accidently removed in #174 by @tomwhite in https://github.com/tomwhite/cubed/pull/185
Improvements and fixes to Modal examples by @tomwhite in https://github.com/tomwhite/cubed/pull/190
More examples on GCF by @tomwhite in https://github.com/tomwhite/cubed/pull/198
Fix measure reserved by @tomwhite in https://github.com/tomwhite/cubed/pull/200
Include 'cubed' in the directories created in the working directory by @tomwhite in https://github.com/tomwhite/cubed/pull/202
Add user guide and break out getting started into separate pages by @tomwhite in https://github.com/tomwhite/cubed/pull/203
Show array variable name in visualization by @tomwhite in https://github.com/tomwhite/cubed/pull/192
Runtime refactoring by @tomwhite in https://github.com/tomwhite/cubed/pull/206
Make Zarr array creation lazy by @tomwhite in https://github.com/tomwhite/cubed/pull/207
Make executors create Zarr arrays by @tomwhite in https://github.com/tomwhite/cubed/pull/209
Make Lithops retries apply per-task by @tomwhite in https://github.com/tomwhite/cubed/pull/211
Move Modal examples to 'aws' directory by @tomwhite in https://github.com/tomwhite/cubed/pull/212
Publish packages to PyPI using GitHub Actions by @tomwhite in https://github.com/tomwhite/cubed/pull/213

Full Changelog: https://github.com/tomwhite/cubed/compare/0.7.0...0.8.0

- Python
Published by tomwhite almost 3 years ago