Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of virtualizarr

virtualizarr - v2.1.2

Patch release with minor bug fixes for the DMRPParser and Icechunk writing behavior.

What's Changed

  • New empty release notes by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/781
  • Minor improvements to dmrpp parser by @Mikejmnez in https://github.com/zarr-developers/VirtualiZarr/pull/757
  • account for endian being possibly None by @keewis in https://github.com/zarr-developers/VirtualiZarr/pull/787
  • Do not validate references for "empty" chunks. by @sharkinsspatial in https://github.com/zarr-developers/VirtualiZarr/pull/791
  • Remove unnecessary dtype conversion in icechunk writer by @popododo0720 in https://github.com/zarr-developers/VirtualiZarr/pull/760
  • Polish release notes for v2.1.2 by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/793

New Contributors

  • @Mikejmnez made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/757
  • @popododo0720 made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/760

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/compare/v2.1.1...v2.1.2

- Python
Published by maxrjones 6 months ago

virtualizarr - v2.1.1

Extremely minor release to ensure compatibility with the soon-to-be released version of xarray (v2025.08.0).

What's Changed

  • Empty release notes by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/778
  • Adjust for small change in signature of private xarray function _nested_combine by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/779
  • Eelease notes for v2.1.1 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/780

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/compare/v2.1.0...v2.1.1

- Python
Published by TomNicholas 7 months ago

virtualizarr - v2.1.0

This release fixes a number of important bugs that could silently lead to referenced data being read back incorrectly. In particular, note that writing virtual chunks to Icechunk now requires that all virtual chunk containers are set correctly by default. It also unpins our dependency on xarray, so that VirtualiZarr is compatible with the latest released version of Xarray. Please upgrade!

What's Changed

  • Add template for empty release notes in release guide by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/755
  • Add empty release notes section by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/754
  • Adds Mkdocs alternate tabs to show multiple store configurations on Usage.md by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/741
  • [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in https://github.com/zarr-developers/VirtualiZarr/pull/756
  • Include serializer from ManifestStore in Icechunk virtual references by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/766
  • Raise instead of silently dropping non-zarr custom codec by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/775
  • Raise if missing virtual chunk containers by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/774
  • Unpin xarray by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/765
  • Extract endianness from Bytes codec in V2 metadata conversion by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/769
  • Note breaking change of enforcing virtual chunk containers are correct by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/776
  • v2.1.0 release notes by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/777

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/compare/v2.0.1...v2.1.0

- Python
Published by TomNicholas 7 months ago

virtualizarr - v2.0.1

Minor release to ensure compatibility with incoming changes to Icechunk.

What's Changed

  • Fix indexing with nones by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/730
  • Add empty release notes section by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/732
  • Refactor indexing by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/734
  • Don't write empty chunks to icechunk by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/745
  • Fix test paths to make upstream Icechunk happy by @paraseba in https://github.com/zarr-developers/VirtualiZarr/pull/752
  • Document recommendation to use repo.save_config by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/746
  • Enable workflow_dispatch for upstream workflow by @paraseba in https://github.com/zarr-developers/VirtualiZarr/pull/751
  • Add comment on skip_signature=True in the s3 store example by @jbusecke in https://github.com/zarr-developers/VirtualiZarr/pull/731
  • v2.0.1 release notes by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/753

New Contributors

  • @paraseba made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/752

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/compare/v2.0.0...v2.0.1

- Python
Published by TomNicholas 7 months ago

virtualizarr - v2.0.0

🎉 v2.0.0 - Second major release of VirtualiZarr 🎉

Highlights

Detailed Release Notes

Read more about the numerous new features, enhancements, bug fixes, and internal improvements in the release notes.

Migration Guide

The migration guide contains detailed information about switching from VirtualiZarr 1.0 to VirtualiZarr 2.0.

- Python
Published by maxrjones 8 months ago

virtualizarr - v2.0.0rc1

VirtualiZarr v2.0.0 Release Candidate 1

Highlights

Detailed Release Notes

Read more about the numerous new features, enhancements, bug fixes, and internal improvements in the release notes.

Migration Guide

The migration guide contains detailed information about switching from VirtualiZarr 1.0 to VirtualiZarr 2.0.

- Python
Published by maxrjones 8 months ago

virtualizarr - v1.3.2

Small release which fixes a problem causing the docs to be out of date, fixes some issues in the tests with unclosed file handles, but also increases the performance of writing large numbers of virtual references to Icechunk!

What's Changed

  • Upgrade icechunk 0.2.0 by @abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/445
  • Update store.set_virtual_ref -> store.set_virtual_refs by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/443
  • Align HDF reader CF _FillValue with Zarr v3 semantics by @sharkinsspatial in https://github.com/zarr-developers/VirtualiZarr/pull/420
  • Revert "Align HDF reader CF _FillValue with Zarr v3 semantics (#420)" by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/457
  • Store -> Datacube language in readme by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/456
  • Close open file handle if there aren't loadable_vars by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/455
  • Reapply HDF reader fillvalue by @chuckwondo in https://github.com/zarr-developers/VirtualiZarr/pull/460
  • Update minimum icechunk version to v0.2.4 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/462
  • Add missing pytest-asyncio plugin by @chuckwondo in https://github.com/zarr-developers/VirtualiZarr/pull/461
  • Release notes for v1.3.2 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/463

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/compare/v1.3.1...v1.3.2

- Python
Published by TomNicholas about 1 year ago

virtualizarr - v1.3.1

This is a minor version that among other things, fixes examples by using the new Icechunk syntax.

What's Changed

  • Install Icechunk from Conda in CI by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/425
  • Remove Zarr chunk manifest writing functionality by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/426
  • Update zarr 3 codec pipeline workflow by @mpiannucci in https://github.com/zarr-developers/VirtualiZarr/pull/423
  • Add in-memory icechunk tests to existing roundtrip tests by @jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/422
  • Updates examples to newer Icechunk syntax by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/431

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/compare/v1.3.0...v1.3.1

- Python
Published by rsignell about 1 year ago

virtualizarr - v1.3.0

This release stabilises our dependencies - you can now use released versions of VirtualiZarr, Kerchunk, and Icechunk all in the same environment!

It also fixes a number of bugs, adds minor features, changes the default reader for HDF/netCDF4 files, and includes refactors to reduce code redundancy with zarr-python v3. You can also choose which sets of dependencies you want at installation time.

What's Changed

  • Release guide release notes correction by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/333
  • Improve readme and top-level package description by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/337
  • Fix group kwarg by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/338
  • Legacy->archival by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/344
  • move some kerchunk tests to dedicated test module by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/351
  • Nit: upstream.yml was named min-deps by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/355
  • Remove warning when passing indexes=None by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/357
  • Add nbytes property by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/227
  • More indexes fixes by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/358
  • Configure codecov to ignore tests folder by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/362
  • xfail test which times out due to large nisar file by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/370
  • Sync with icechunk alpha 8 by @mpiannucci in https://github.com/zarr-developers/VirtualiZarr/pull/368
  • [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/372
  • Test FITS reader on Hubble Space Telescope data by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/363
  • Align accepted filetypes with docstring description by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/361
  • Keep fill value for encoding by @abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/369
  • Fix hdf typing by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/380
  • Fix subgroup dims in HDF reader by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/366
  • Add group option to dataset_to_icechunk by @chuckwondo in https://github.com/zarr-developers/VirtualiZarr/pull/383
  • Silence pip root user warnings in CI by @chuckwondo in https://github.com/zarr-developers/VirtualiZarr/pull/384
  • Address warnings emitted during test execution for deallocating cachefilemanager by @chuckwondo in https://github.com/zarr-developers/VirtualiZarr/pull/386
  • Add group param to virtualize Dataset accessor by @chuckwondo in https://github.com/zarr-developers/VirtualiZarr/pull/391
  • Ignore conftest.py in code coverage by @chuckwondo in https://github.com/zarr-developers/VirtualiZarr/pull/388
  • Fix drop_variables support in kerchunk, dmrpp by @chuckwondo in https://github.com/zarr-developers/VirtualiZarr/pull/396
  • Skip tests that often take ages in the CI by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/398
  • Updated docs for loading variables by @kjdoore in https://github.com/zarr-developers/VirtualiZarr/pull/334
  • Vendor kerchunk netCDF3 reader by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/397
  • Switch to custom netcdf4/hdf5 backend by @jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/395
  • Revert "Vendor kerchunk netCDF3 reader (#397)" by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/408
  • Update usage.md with append method for icechunk by @abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/409
  • Fix HDFVirtualBackend handling of non coordinate dimension HDF datasets. by @sharkinsspatial in https://github.com/zarr-developers/VirtualiZarr/pull/410
  • Update to zarr 3 and main kerchunk by @jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/406
  • Update to zarr-v3-compatible released version of Kerchunk (0.2.8) by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/412
  • Split optional dependencies in pyproject.toml by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/309
  • Fix tests by recreating the repo every time by @abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/417
  • [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/416
  • Update Icechunk dependency to v0.1.1 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/419

New Contributors

  • @chuckwondo made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/383
  • @kjdoore made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/334

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/compare/v1.2.0...v1.3.0

- Python
Published by TomNicholas about 1 year ago

virtualizarr - v1.2.0

This release brings a stricter internal model for manifest paths, support for appending to existing icechunk stores, an experimental non-kerchunk-based HDF5 reader, handling of nested groups in DMR++ files, as well as many other bugfixes and documentation improvements.

What's Changed

  • FAQ updates by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/266
  • Import top-level version of xarray classes by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/267
  • Update index.md by @thodson-usgs in https://github.com/zarr-developers/VirtualiZarr/pull/275
  • Remove unused ManifestBackendArray class by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/282
  • Fix bug in RT of parquet detection by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/278
  • Search for coordnames in separatecoords by @ayushnag in https://github.com/zarr-developers/VirtualiZarr/pull/191
  • [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/283
  • Bump minimum Xarray dependency to 2024.10.0 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/284
  • Dont write ARRAYDIMENSIONS to icechunk by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/286
  • Fix release notes for v1.1.0 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/288
  • dmrpp root and nested group parsing fix by @ayushnag in https://github.com/zarr-developers/VirtualiZarr/pull/265
  • Update README.md by @joshmoore in https://github.com/zarr-developers/VirtualiZarr/pull/294
  • Remove numcodecs specific install by @mpiannucci in https://github.com/zarr-developers/VirtualiZarr/pull/301
  • Update contributors guide by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/298
  • Clarify which features are currently available in FAQ by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/296
  • Fix sphinx warnings by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/300
  • Update pkg install in docs contribution guide by @douglatornell in https://github.com/zarr-developers/VirtualiZarr/pull/304
  • Support downstream type checking by including py.typed by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/306
  • Non-kerchunk backend for HDF5/netcdf4 files. by @sharkinsspatial in https://github.com/zarr-developers/VirtualiZarr/pull/87
  • Release note for #87 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/307
  • Add status badges to README by @douglatornell in https://github.com/zarr-developers/VirtualiZarr/pull/303
  • Expand xarray openable type hint with ReadBuffer by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/316
  • Consolidate hdf reader tests into their own tests module by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/314
  • Refactor kerchunk reader tests to call openvirtualdataset by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/317
  • Add virtualbackendkwargs argument to openvirtualdataset by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/315
  • [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/322
  • Refactor dmrpp tests to expose data file path by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/323
  • Paths as URIs by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/243
  • Add list of previous talks to readme by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/313
  • Correct some documentation links to function the public API by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/325
  • Ignore spontaneous mypy errors for h5py classes. ref #324 by @sharkinsspatial in https://github.com/zarr-developers/VirtualiZarr/pull/328
  • Append to icechunk stores by @abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/272
  • Add new page with links to example notebooks by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/331
  • Release summary for v1.2.0 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/332

New Contributors

  • @joshmoore made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/294
  • @douglatornell made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/304
  • @sharkinsspatial made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/87

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/compare/v1.1.0...v1.2.0

- Python
Published by TomNicholas over 1 year ago

virtualizarr - v1.1.0

This release adds Icechunk support!!

It also brings a complete refactoring of the system of readers and writers internally, which allowed us to make Kerchunk an optional dependency. There are also many other bugfixes and smaller improvements.

What's Changed

  • xr.testing with ManifestArray fix (update isnan ufunc) by @ayushnag in https://github.com/zarr-developers/VirtualiZarr/pull/188
  • Clarify that virtualizarr is a user-level replacement for kerchunk by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/192
  • Exclude empty paths on ChunkDict creation by @ghidalgo3 in https://github.com/zarr-developers/VirtualiZarr/pull/198
  • Extend refspec support to [path] entries (without offset/length) by @maresb in https://github.com/zarr-developers/VirtualiZarr/pull/187
  • Conformant ZarrV3 codecs and fill values by @ghidalgo3 in https://github.com/zarr-developers/VirtualiZarr/pull/193
  • https access fix by @ayushnag in https://github.com/zarr-developers/VirtualiZarr/pull/196
  • Handle scalar dataset variables by @ghidalgo3 in https://github.com/zarr-developers/VirtualiZarr/pull/205
  • Set ZArray default fill_value as NaT for datetime64 by @thodson-usgs in https://github.com/zarr-developers/VirtualiZarr/pull/206
  • Update .pre-commit-config mypy + bump ruff version by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/211
  • Update static typing by @TomAugspurger in https://github.com/zarr-developers/VirtualiZarr/pull/213
  • Adds concurrency to CI w/ cancel-in-progress=True by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/214
  • Implement pydantic models as dataclasses by @TomAugspurger in https://github.com/zarr-developers/VirtualiZarr/pull/210
  • use the theme options for pydata_sphinx_theme by @keewis in https://github.com/zarr-developers/VirtualiZarr/pull/223
  • Removes default storage options by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/228
  • openvirtualdataset with dmr++ by @ayushnag in https://github.com/zarr-developers/VirtualiZarr/pull/113
  • Internal refactor to separate reading and writing concerns by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/231
  • Let Xarray handle decode_times by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/232
  • Support specifying single HDF Group in openvirtualdataset by @scottyhq in https://github.com/zarr-developers/VirtualiZarr/pull/165
  • Adds defaults in open_virtual_dataset_from_v3_store by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/234
  • Virtualizarr + Coiled Serverless Example Notebook by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/233
  • [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/236
  • Add example to create a virtual dataset using lithops by @thodson-usgs in https://github.com/zarr-developers/VirtualiZarr/pull/203
  • Update backend.py (tiny typo) by @mdsumner in https://github.com/zarr-developers/VirtualiZarr/pull/240
  • Makes mypy a seperate CI job by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/254
  • Fix mypy errors around numpy functions not being strictly type hinted by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/252
  • Allow open_virtual_dataset to read existing Kerchunk references by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/251
  • Skip tests that require kerchunk by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/259
  • allow creating references for empty archival datasets by @keewis in https://github.com/zarr-developers/VirtualiZarr/pull/260
  • Split kerchunk reader up by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/261
  • [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/250
  • Add CI job for testing upstream versions of dependencies by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/264
  • Add Icechunk Support by @mpiannucci in https://github.com/zarr-developers/VirtualiZarr/pull/256

New Contributors

  • @ayushnag made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/188
  • @ghidalgo3 made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/198
  • @maresb made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/187
  • @thodson-usgs made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/206
  • @keewis made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/223
  • @mdsumner made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/240
  • @mpiannucci made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/256

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/compare/v1.0.0...v1.1.0

- Python
Published by TomNicholas over 1 year ago

virtualizarr - v1.0.0

This release marks VirtualiZarr as mostly feature-complete, in the sense of achieving feature parity with kerchunk's logic for combining datasets, providing an easier way to manipulate kerchunk references in memory and generate kerchunk reference files on disk.

Future VirtualiZarr development will focus on generalizing and upstreaming useful concepts into the Zarr specification, the Zarr-Python library, Xarray, and possibly some new packages. See the roadmap in the documentation for details.

What's Changed

  • Hypothesis test broadcasting by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/139
  • Empty release notes for v0.2 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/145
  • Mark tests which require network access by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/144
  • Install dependencies for tests via mamba by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/148
  • Use default version scheme for setuptools_scm by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/149
  • Use 3 numpy arrays for manifest internally by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/107
  • Rename paths in manifest by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/152
  • Ensure ARRAYDIMENSIONS get dropped from attrs by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/153
  • Ensure attributes on coordinate variables are preserved during round-tripping by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/154
  • Identify non dimension coords by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/156
  • Also test exporting references to in-memory kerchunk reference dict by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/158
  • Use magic bytes to identify file formats by @scottyhq in https://github.com/zarr-developers/VirtualiZarr/pull/143
  • Decoding cftime_variables by @jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/122
  • Fix opening tiff and fits files by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/162
  • Clarify that virtual datasets are not normal xarray datasets by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/173
  • Warn on index creation by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/170
  • Update roadmap for v1.0 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/164
  • Add example of using cftime_variables to usage docs by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/174
  • Future-proof offset and size records in chunkmanifest by @moradology in https://github.com/zarr-developers/VirtualiZarr/pull/177
  • Use a set to avoid duplicate var names from kerchunk by @moradology in https://github.com/zarr-developers/VirtualiZarr/pull/179
  • v1.0 release notes by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/181

New Contributors

  • @scottyhq made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/143
  • @moradology made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/177

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/compare/v0.1.0...v1.0

- Python
Published by TomNicholas over 1 year ago

virtualizarr - v0.1.0

The first release of VirtualiZarr!

This release presents the basic MVP of this library, including the ability to inspect netCDF4/HDF5 files, store the byte ranges in an xarray.Dataset via ManifestArray objects, concatenate those objects, then serialize the result to disk as kerchunk-formatted reference files.

Expect more features and significant optimizations soon.

What's Changed

  • Xarray accessor to create kerchunk reference dict by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/28
  • Equality checking by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/30
  • Support xarray concat, including broadcasting by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/34
  • CI for running tests by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/36
  • Roughout of Sphinx Docs by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/27
  • Updated doc.yml to include pip by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/40
  • Adds netCDF3 vs netCDF4 distinction to automaticallydetermine_filetype. by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/43
  • Test concat of dimension coordinate not backed by an index by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/44
  • Rename opendatasetviakerchunk to openvirtual_dataset by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/47
  • Narrative docs by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/48
  • More narrative docs by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/50
  • Update CI checks with ruff by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/54
  • openvirtualdataset with and without indexes by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/52
  • API docs by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/56
  • Updated NetCDF IO path by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/55
  • Created conftest.py and moved two fixtures into conftest by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/57
  • Ab/filters dtype by @abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/66
  • Switching netcdf3 & netcdf4 filetype detection to file magic 🧙 by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/64
  • [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/70
  • Load selected variables instead of making them virtual by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/69
  • pinkerchunk0.2.2 by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/75
  • Remove python 3.12 from CI matrix by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/76
  • Convert user defined filetype to FileType by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/79
  • FAQ page by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/81
  • Try to remove sidebar in docs by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/82
  • main.yml CI installs from pyproject.toml by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/90
  • Update installation instructions by @jbusecke in https://github.com/zarr-developers/VirtualiZarr/pull/91
  • Write manifests to zarr store by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/45
  • [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/101
  • Bump Ruff version and add formatting by @jbusecke in https://github.com/zarr-developers/VirtualiZarr/pull/98
  • Opening 0D scalars by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/102
  • Fix bug with expand dims of a scalar array by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/103
  • Install xarray from main by @jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/106
  • Unpin kerchunk (set floor) and enable Python 3.12 by @jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/108
  • Depend on latest version of xarray by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/109
  • Remove python 3.9 by @jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/112
  • Adding reader_options kwargs to openvirtualdataset. by @norlandrhagen in https://github.com/zarr-developers/VirtualiZarr/pull/67
  • Write to parquet by @jsignell in https://github.com/zarr-developers/VirtualiZarr/pull/110
  • Test fsspec roundtrip by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/42
  • Inline loaded variables into kerchunk references by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/73
  • Release notes page by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/120
  • requires-python = ">=3.10" by @abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/127
  • Pass args and add test by @abarciauskas-bgse in https://github.com/zarr-developers/VirtualiZarr/pull/128
  • [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/zarr-developers/VirtualiZarr/pull/129
  • Allow other fsspec protocols than local and s3 by @TomAugspurger in https://github.com/zarr-developers/VirtualiZarr/pull/126
  • Add dunder version to top-level init.py by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/133
  • Add release workflow by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/136
  • Only run distribution workflow on releases by @maxrjones in https://github.com/zarr-developers/VirtualiZarr/pull/140
  • change default reader_options to None by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/137
  • Replace np.NaN with np.nan in preparation for numpy 2.0 by @TomNicholas in https://github.com/zarr-developers/VirtualiZarr/pull/138

New Contributors

  • @TomNicholas made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/28
  • @abarciauskas-bgse made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/66
  • @pre-commit-ci made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/70
  • @jbusecke made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/91
  • @jsignell made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/106
  • @TomAugspurger made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/126
  • @maxrjones made their first contribution in https://github.com/zarr-developers/VirtualiZarr/pull/133

Full Changelog: https://github.com/zarr-developers/VirtualiZarr/commits/v0.1

- Python
Published by TomNicholas over 1 year ago