Releases | Open Source Science

https://github.com/lancedb/lance - v0.35.0-beta.2

What's Changed

New Features 🎉

feat: build Linux wheels with debug symbols for GitHub releases by @wjones127 in https://github.com/lancedb/lance/pull/4647

Full Changelog: https://github.com/lancedb/lance/compare/v0.35.0-beta.1...v0.35.0-beta.2

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.35.0-beta.1

What's Changed

New Features 🎉

feat: build type-aware index for JSON by @Xuanwo in https://github.com/lancedb/lance/pull/4626
feat: add json parser for FtsQuery by @wojiaodoubao in https://github.com/lancedb/lance/pull/4605 ### Documentation 📚
docs: polish docs for JSON data types by @Xuanwo in https://github.com/lancedb/lance/pull/4640 ### Other Changes
refactor(python): refactor take bench and support parametrize compression by @yanghua in https://github.com/lancedb/lance/pull/4636
refactor: remove the 2.2 version requirement for using JSON features by @Xuanwo in https://github.com/lancedb/lance/pull/4641

Full Changelog: https://github.com/lancedb/lance/compare/v0.35.0...v0.35.0-beta.1

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.35.0

What's Changed

Breaking Changes 🛠

feat!: add storage options to wrapping object store by @wkalt in https://github.com/lancedb/lance/pull/4509
feat!: shallow_clone supports index by @majin1102 in https://github.com/lancedb/lance/pull/4553 ### New Features 🎉
feat: add lance-tools command by @cmccabe in https://github.com/lancedb/lance/pull/4545
feat: add JSONB read/write support by @Xuanwo in https://github.com/lancedb/lance/pull/4566
feat(java): add deleteRows for Fragment by @fangbo in https://github.com/lancedb/lance/pull/4528
feat: inverted index for contains_tokens by @wojiaodoubao in https://github.com/lancedb/lance/pull/4489
feat: add cumulativecpu output to analyzeplan string by @cmccabe in https://github.com/lancedb/lance/pull/4519
feat(rust): support update stable row id for overlapping by @yanghua in https://github.com/lancedb/lance/pull/4496
feat: add UDFs for json by @Xuanwo in https://github.com/lancedb/lance/pull/4577
feat: allow using opendal to access s3, azblob and gcs by @jackye1995 in https://github.com/lancedb/lance/pull/4597
feat: expose open_session in python by @wojiaodoubao in https://github.com/lancedb/lance/pull/4581
feat: add a scalar index for JSON by @westonpace in https://github.com/lancedb/lance/pull/4621 ### Bug Fixes 🐛
fix: include base_id in DeletionFile serialization by @pimdh in https://github.com/lancedb/lance/pull/4580
fix: taggedoldversions should only track tagged and old versions by @wojiaodoubao in https://github.com/lancedb/lance/pull/4592
fix: index out of bounds caused by exhausted posting iterator by @BubbleCal in https://github.com/lancedb/lance/pull/4587
fix: validate operations in transaction commit by @majin1102 in https://github.com/lancedb/lance/pull/4532
fix: disable default features from jsonb to avoid changing serde-json behavior by @Xuanwo in https://github.com/lancedb/lance/pull/4601
fix: shallow_clone multiple times refering to wrong base path by @majin1102 in https://github.com/lancedb/lance/pull/4617
fix: can't train vector index with cosine distance on GPU by @BubbleCal in https://github.com/lancedb/lance/pull/4623 ### Documentation 📚
docs: add docs for JSON and JSON UDFs by @Xuanwo in https://github.com/lancedb/lance/pull/4599 ### Performance Improvements 🚀
perf: add dataset random take benchmark by @yanghua in https://github.com/lancedb/lance/pull/4113
perf: improve FTS performance for long query by @BubbleCal in https://github.com/lancedb/lance/pull/4576 ### Other Changes
refactor: rework scalar index loading, training, and parsing into a plugin trait by @westonpace in https://github.com/lancedb/lance/pull/4584

New Contributors

@pimdh made their first contribution in https://github.com/lancedb/lance/pull/4580
@ebyhr made their first contribution in https://github.com/lancedb/lance/pull/4606

Full Changelog: https://github.com/lancedb/lance/compare/v0.34.0...v0.35.0

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.34.0-beta.4

What's Changed

Breaking Changes 🛠

feat!: add storage options to wrapping object store by @wkalt in https://github.com/lancedb/lance/pull/4509
feat!: shallow_clone supports index by @majin1102 in https://github.com/lancedb/lance/pull/4553 ### New Features 🎉
feat: add lance-tools command by @cmccabe in https://github.com/lancedb/lance/pull/4545
feat: add JSONB read/write support by @Xuanwo in https://github.com/lancedb/lance/pull/4566
feat(java): add deleteRows for Fragment by @fangbo in https://github.com/lancedb/lance/pull/4528
feat: inverted index for contains_tokens by @wojiaodoubao in https://github.com/lancedb/lance/pull/4489
feat: add cumulativecpu output to analyzeplan string by @cmccabe in https://github.com/lancedb/lance/pull/4519
feat(rust): support update stable row id for overlapping by @yanghua in https://github.com/lancedb/lance/pull/4496
feat: add UDFs for json by @Xuanwo in https://github.com/lancedb/lance/pull/4577
feat: allow using opendal to access s3, azblob and gcs by @jackye1995 in https://github.com/lancedb/lance/pull/4597 ### Bug Fixes 🐛
fix: include base_id in DeletionFile serialization by @pimdh in https://github.com/lancedb/lance/pull/4580
fix: taggedoldversions should only track tagged and old versions by @wojiaodoubao in https://github.com/lancedb/lance/pull/4592
fix: index out of bounds caused by exhausted posting iterator by @BubbleCal in https://github.com/lancedb/lance/pull/4587
fix: validate operations in transaction commit by @majin1102 in https://github.com/lancedb/lance/pull/4532
fix: disable default features from jsonb to avoid changing serde-json behavior by @Xuanwo in https://github.com/lancedb/lance/pull/4601
fix: shallow_clone multiple times refering to wrong base path by @majin1102 in https://github.com/lancedb/lance/pull/4617 ### Documentation 📚
docs: add docs for JSON and JSON UDFs by @Xuanwo in https://github.com/lancedb/lance/pull/4599 ### Performance Improvements 🚀
perf: add dataset random take benchmark by @yanghua in https://github.com/lancedb/lance/pull/4113
perf: improve FTS performance for long query by @BubbleCal in https://github.com/lancedb/lance/pull/4576 ### Other Changes
refactor: rework scalar index loading, training, and parsing into a plugin trait by @westonpace in https://github.com/lancedb/lance/pull/4584

New Contributors

@pimdh made their first contribution in https://github.com/lancedb/lance/pull/4580
@ebyhr made their first contribution in https://github.com/lancedb/lance/pull/4606

Full Changelog: https://github.com/lancedb/lance/compare/v0.34.0...v0.34.0-beta.4

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.34.0

What's Changed

Breaking Changes 🛠

refactor!: rename stable row id feature flag by @yanghua in https://github.com/lancedb/lance/pull/4481
feat!: support shallow_clone in dataset by @majin1102 in https://github.com/lancedb/lance/pull/4257 ### New Features 🎉
feat(java): java transaction supports DataReplacement operation by @majin1102 in https://github.com/lancedb/lance/pull/4454
feat: add contains_tokens udf by @wojiaodoubao in https://github.com/lancedb/lance/pull/4420
feat: add retry-based conflict resolution for delete operations (#4159) by @wjones127 in https://github.com/lancedb/lance/pull/4292
feat: add fuzz tests for encoding by @Xuanwo in https://github.com/lancedb/lance/pull/4480
feat: implement large read io split in format 2.1 by @Xuanwo in https://github.com/lancedb/lance/pull/4462
feat(python): expose diff_meta for pylance by @yanghua in https://github.com/lancedb/lance/pull/4502
feat: add automatic conflict resolution support for delete operations by @wjones127 in https://github.com/lancedb/lance/pull/4407
feat: add a cycle_bool data generator by @westonpace in https://github.com/lancedb/lance/pull/4507
feat: allow reading the _rowoffset by @westonpace in https://github.com/lancedb/lance/pull/4478
feat: add zonemap filter support to Lance at table level by @HaochengLIU in https://github.com/lancedb/lance/pull/4244
feat(rust): support update stable row id for merge insert without index by @yanghua in https://github.com/lancedb/lance/pull/4450
fix: handle parsing sqlcastexpr to datafusion expr properly by @imededin in https://github.com/lancedb/lance/pull/4541
feat: provide feature flag for shallow cloning by @majin1102 in https://github.com/lancedb/lance/pull/4552
feat: add lance file version 2.2 by @Xuanwo in https://github.com/lancedb/lance/pull/4568
feat: add more metrics for FTS queries by @BubbleCal in https://github.com/lancedb/lance/pull/4560 ### Bug Fixes 🐛
fix: lance failed to encode 1025 rep/def values by @Xuanwo in https://github.com/lancedb/lance/pull/4467
fix: ensure transaction rebase happens in ascending order by @jackye1995 in https://github.com/lancedb/lance/pull/4486
fix: posting lists cache conflicts across partitions by @BubbleCal in https://github.com/lancedb/lance/pull/4490
fix: the elapsed_time is not correct for IvfSubIndex Node by @BubbleCal in https://github.com/lancedb/lance/pull/4491
fix: skip rebasing for strict overwrite by @HubertY in https://github.com/lancedb/lance/pull/4377
fix: use LanceBuffer::borrowtotyped_slice for safe slice cast by @Xuanwo in https://github.com/lancedb/lance/pull/4493
fix: avoid column name collision in mergeinsert by using _action by @wjones127 in https://github.com/lancedb/lance/pull/4499
fix: remove data files with only tombstoned fields by @wjones127 in https://github.com/lancedb/lance/pull/4497
fix: address fuzz test #4492 by @Xuanwo in https://github.com/lancedb/lance/pull/4503
fix: shrinktofit after copyfrombytes to ensure memory size is correct by @Xuanwo in https://github.com/lancedb/lance/pull/4515
fix: fix FTS index cache underutilization due to DeepSizeOf overestimation by @LuQQiu in https://github.com/lancedb/lance/pull/4513
fix: full text search BM25 score is not accurate by @BubbleCal in https://github.com/lancedb/lance/pull/4525
fix: use java/lang/String to get DeletionFileType value by @fangbo in https://github.com/lancedb/lance/pull/4526
fix: incorrect logic for logical or of RowIdMask by @westonpace in https://github.com/lancedb/lance/pull/4531
fix: fix spelling in comment in dataset.rs by @cmccabe in https://github.com/lancedb/lance/pull/4554
fix: restore behavior of projection plans from schema by @westonpace in https://github.com/lancedb/lance/pull/4562
fix: get rust-benchmark.yml working again by @cmccabe in https://github.com/lancedb/lance/pull/4567
fix(doc): fix take_blobs python doc wrong parameters format by @yanghua in https://github.com/lancedb/lance/pull/4569 ### Documentation 📚
docs: fix inverted index default values in pydoc by @chenghao-guo in https://github.com/lancedb/lance/pull/4565 ### Performance Improvements 🚀
perf: use heap to maintain topk results by @BubbleCal in https://github.com/lancedb/lance/pull/4505
perf: remove tracing instrumentation from FTS hot paths by @LuQQiu in https://github.com/lancedb/lance/pull/4536
perf: fallback to brute force FTS if filters matching fewer rows by @BubbleCal in https://github.com/lancedb/lance/pull/4551
perf: skip more documents in FTS flat search by @BubbleCal in https://github.com/lancedb/lance/pull/4570 ### Other Changes
refactor: remove owned LanceBuffer to eliminate runtime try_clone checks by @Xuanwo in https://github.com/lancedb/lance/pull/4460
refactor: reduce lance-encoding compile times by @westonpace in https://github.com/lancedb/lance/pull/4474
refactor: remove special case in cancasttypes by @cmccabe in https://github.com/lancedb/lance/pull/4549
refactor: remove Ray integration from Lance core by @jackye1995 in https://github.com/lancedb/lance/pull/4563

New Contributors

@imededin made their first contribution in https://github.com/lancedb/lance/pull/4541
@cmccabe made their first contribution in https://github.com/lancedb/lance/pull/4549

Full Changelog: https://github.com/lancedb/lance/compare/v0.33.0...v0.34.0

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.33.0-beta.3

What's Changed

New Features 🎉

feat: implement large read io split in format 2.1 by @Xuanwo in https://github.com/lancedb/lance/pull/4462
feat(python): expose diff_meta for pylance by @yanghua in https://github.com/lancedb/lance/pull/4502
feat: add automatic conflict resolution support for delete operations by @wjones127 in https://github.com/lancedb/lance/pull/4407 ### Bug Fixes 🐛
fix: avoid column name collision in mergeinsert by using _action by @wjones127 in https://github.com/lancedb/lance/pull/4499
fix: remove data files with only tombstoned fields by @wjones127 in https://github.com/lancedb/lance/pull/4497
fix: address fuzz test #4492 by @Xuanwo in https://github.com/lancedb/lance/pull/4503 ### Performance Improvements 🚀
perf: use heap to maintain topk results by @BubbleCal in https://github.com/lancedb/lance/pull/4505

Full Changelog: https://github.com/lancedb/lance/compare/v0.33.0-beta.2...v0.33.0-beta.3

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.33.0-beta.2

What's Changed

New Features 🎉

feat: add contains_tokens udf by @wojiaodoubao in https://github.com/lancedb/lance/pull/4420
feat: add retry-based conflict resolution for delete operations (#4159) by @wjones127 in https://github.com/lancedb/lance/pull/4292
feat: add fuzz tests for encoding by @Xuanwo in https://github.com/lancedb/lance/pull/4480 ### Bug Fixes 🐛
fix: lance failed to encode 1025 rep/def values by @Xuanwo in https://github.com/lancedb/lance/pull/4467
fix: ensure transaction rebase happens in ascending order by @jackye1995 in https://github.com/lancedb/lance/pull/4486
fix: posting lists cache conflicts across partitions by @BubbleCal in https://github.com/lancedb/lance/pull/4490
fix: the elapsed_time is not correct for IvfSubIndex Node by @BubbleCal in https://github.com/lancedb/lance/pull/4491
fix: skip rebasing for strict overwrite by @HubertY in https://github.com/lancedb/lance/pull/4377
fix: use LanceBuffer::borrowtotyped_slice for safe slice cast by @Xuanwo in https://github.com/lancedb/lance/pull/4493

Full Changelog: https://github.com/lancedb/lance/compare/v0.33.0-beta.1...v0.33.0-beta.2

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.33.0-beta.1

What's Changed

New Features 🎉

feat(java): java transaction supports DataReplacement operation by @majin1102 in https://github.com/lancedb/lance/pull/4454 ### Other Changes
refactor: remove owned LanceBuffer to eliminate runtime try_clone checks by @Xuanwo in https://github.com/lancedb/lance/pull/4460

Full Changelog: https://github.com/lancedb/lance/compare/v0.33.0...v0.33.0-beta.1

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.33.0

What's Changed

Breaking Changes 🛠

feat(python)!: unify Python CreateIndex api with Rust implementation by @chenghao-guo in https://github.com/lancedb/lance/pull/4438
2.1 format has breaking changes in this version. Files written with earlier versions may not be readable.

New Features 🎉

feat: allow creating empty scalar indices by @wjones127 in https://github.com/lancedb/lance/pull/4033
feat(python): release GIL in dataset API by @lorinlee in https://github.com/lancedb/lance/pull/4389
feat: expose decoder config at python side by @Xuanwo in https://github.com/lancedb/lance/pull/4335
feat: allow empty projection by @westonpace in https://github.com/lancedb/lance/pull/4340
feat(java): dataset supports reading transaction by @majin1102 in https://github.com/lancedb/lance/pull/4397
feat: implement explain_plan and analyze_plan for merge_insert operations by @wjones127 in https://github.com/lancedb/lance/pull/4295
feat(rust): support diff_meta api by @yanghua in https://github.com/lancedb/lance/pull/4404
feat: auto-build Lance namespace and Spark doc in main Lance website by @jackye1995 in https://github.com/lancedb/lance/pull/4411
feat: allow row ids (offsets, addresses) in filters to be translated into an optimized take op by @westonpace in https://github.com/lancedb/lance/pull/4394
feat(java): java transaction supports merge operation by @majin1102 in https://github.com/lancedb/lance/pull/4419
feat(java): transaction supports UpdateConfig operation by @majin1102 in https://github.com/lancedb/lance/pull/4406
feat: allow configuring bss for encoding by @Xuanwo in https://github.com/lancedb/lance/pull/4393
feat(java): java transaction supports rewrite operation by @majin1102 in https://github.com/lancedb/lance/pull/4414
feat(java): java transaction support restore and reserveFragments operation by @majin1102 in https://github.com/lancedb/lance/pull/4435
feat(java): introduce update in transaction by @fangbo in https://github.com/lancedb/lance/pull/4408
feat: support blob encoding in format 2.1 by @Xuanwo in https://github.com/lancedb/lance/pull/4449
feat(java): introduce delete in transaction by @fangbo in https://github.com/lancedb/lance/pull/4441
feat(rust): support update stable rowid without index by @yanghua in https://github.com/lancedb/lance/pull/4388 ### Bug Fixes 🐛
fix: limit the depth of filter expression to 500 by @BubbleCal in https://github.com/lancedb/lance/pull/4403
fix: make sure aws feature is applied to code correctly by @vlovich in https://github.com/lancedb/lance/pull/4398
fix: fix target alignment incorrect by using LanceBuffer borrowtotyped_slice by @Xuanwo in https://github.com/lancedb/lance/pull/4413
fix: wrong OSS path append by @adrian-wang in https://github.com/lancedb/lance/pull/4424
fix: too large data chunk generated by highly compressed yet nested data with RLE by @Xuanwo in https://github.com/lancedb/lance/pull/4431
fix: remove explain in sql by @wojiaodoubao in https://github.com/lancedb/lance/pull/4428
fix: panic message when LanceFileWriter is not closed and then got garbage collected by @wojiaodoubao in https://github.com/lancedb/lance/pull/4128
fix: update broken links by @ddupg in https://github.com/lancedb/lance/pull/4447
fix: fix bss been enabled in wrong by @Xuanwo in https://github.com/lancedb/lance/pull/4444
fix: none ASCII path issue by @majin1102 in https://github.com/lancedb/lance/pull/4442
fix: btree prewarm causes offset overflow by @BubbleCal in https://github.com/lancedb/lance/pull/4448
fix: do not panic when reading slices of bitmap column in 2.0 reader by @westonpace in https://github.com/lancedb/lance/pull/4452 ### Performance Improvements 🚀
perf: introduce assume for generating better code by @BubbleCal in https://github.com/lancedb/lance/pull/4439 ### Other Changes
refactor: optimize MiniBlockRepIndex decode by decoding from bytes directly by @Xuanwo in https://github.com/lancedb/lance/pull/4440
refactor: refactor encoding proto for 2.1 to cleanup and future-proof by @westonpace in https://github.com/lancedb/lance/pull/4434

New Contributors

@adrian-wang made their first contribution in https://github.com/lancedb/lance/pull/4424

Full Changelog: https://github.com/lancedb/lance/compare/v0.32.1...v0.33.0

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.32.2-beta.3

What's Changed

New Features 🎉

feat(java): java transaction supports rewrite operation by @majin1102 in https://github.com/lancedb/lance/pull/4414
feat(java): java transaction support restore and reserveFragments operation by @majin1102 in https://github.com/lancedb/lance/pull/4435
feat(java): introduce update in transaction by @fangbo in https://github.com/lancedb/lance/pull/4408 ### Bug Fixes 🐛
fix: too large data chunk generated by highly compressed yet nested data with RLE by @Xuanwo in https://github.com/lancedb/lance/pull/4431

Full Changelog: https://github.com/lancedb/lance/compare/v0.32.2-beta.2...v0.32.2-beta.3

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.32.2-beta.2

What's Changed

New Features 🎉

feat(rust): support diff_meta api by @yanghua in https://github.com/lancedb/lance/pull/4404
feat: auto-build Lance namespace and Spark doc in main Lance website by @jackye1995 in https://github.com/lancedb/lance/pull/4411
feat: allow row ids (offsets, addresses) in filters to be translated into an optimized take op by @westonpace in https://github.com/lancedb/lance/pull/4394
feat(java): java transaction supports merge operation by @majin1102 in https://github.com/lancedb/lance/pull/4419
feat(java): transaction supports UpdateConfig operation by @majin1102 in https://github.com/lancedb/lance/pull/4406
feat: allow configuring bss for encoding by @Xuanwo in https://github.com/lancedb/lance/pull/4393 ### Bug Fixes 🐛
fix: fix target alignment incorrect by using LanceBuffer borrowtotyped_slice by @Xuanwo in https://github.com/lancedb/lance/pull/4413
fix: wrong OSS path append by @adrian-wang in https://github.com/lancedb/lance/pull/4424

New Contributors

@adrian-wang made their first contribution in https://github.com/lancedb/lance/pull/4424

Full Changelog: https://github.com/lancedb/lance/compare/v0.32.2-beta.1...v0.32.2-beta.2

- Rust
Published by wjones127 6 months ago

https://github.com/lancedb/lance - v0.32.2-beta.1

What's Changed

New Features 🎉

feat(python): release GIL in dataset API by @lorinlee in https://github.com/lancedb/lance/pull/4389
feat: expose decoder config at python side by @Xuanwo in https://github.com/lancedb/lance/pull/4335
feat: allow empty projection by @westonpace in https://github.com/lancedb/lance/pull/4340
feat(java): dataset supports reading transaction by @majin1102 in https://github.com/lancedb/lance/pull/4397
feat: implement explain_plan and analyze_plan for merge_insert operations by @wjones127 in https://github.com/lancedb/lance/pull/4295 ### Bug Fixes 🐛
fix: limit the depth of filter expression to 500 by @BubbleCal in https://github.com/lancedb/lance/pull/4403
fix: make sure aws feature is applied to code correctly by @vlovich in https://github.com/lancedb/lance/pull/4398

Full Changelog: https://github.com/lancedb/lance/compare/v0.32.1...v0.32.2-beta.1

- Rust
Published by wjones127 7 months ago

https://github.com/lancedb/lance - v0.32.1

What's Changed

New Features 🎉

feat(java): brings transaction api to Java module and support project by @majin1102 in https://github.com/lancedb/lance/pull/4219
feat: add loading trace event by @yanghua in https://github.com/lancedb/lance/pull/4298
feat: add file reader options in ReadParams by @Xuanwo in https://github.com/lancedb/lance/pull/4287
feat: add matrix type by @BubbleCal in https://github.com/lancedb/lance/pull/4322
feat(java): expose sql api for java api by @fangbo in https://github.com/lancedb/lance/pull/4328
feat: add compression override support for encoding by @Xuanwo in https://github.com/lancedb/lance/pull/4310
feat: add sdk_version in tracing events by @ddupg in https://github.com/lancedb/lance/pull/4347
feat: add transaction properties for commit message handling by @chenghao-guo in https://github.com/lancedb/lance/pull/4078
feat: allow configure encoding via field metadata by @Xuanwo in https://github.com/lancedb/lance/pull/4354
feat: add FragmentSession by @wojiaodoubao in https://github.com/lancedb/lance/pull/4343
feat(java): support configure transaction properties by @majin1102 in https://github.com/lancedb/lance/pull/4363
feat(python): release GIL in LanceFileReader & LanceFileWriter by @lorinlee in https://github.com/lancedb/lance/pull/4383
feat: add verify for encoding roundtrip tests by @Xuanwo in https://github.com/lancedb/lance/pull/4380
feat(java): introduce overwrite and append in transaction by @majin1102 in https://github.com/lancedb/lance/pull/4327
feat: support nullability in struct correctly by @Xuanwo in https://github.com/lancedb/lance/pull/4391 ### Bug Fixes 🐛
fix: add hnsw range search support by @LeoReeYang in https://github.com/lancedb/lance/pull/4311
fix: use debug format for object store provider errors by @jackye1995 in https://github.com/lancedb/lance/pull/4313
fix: update missing lance perf asset by @emmanuel-ferdman in https://github.com/lancedb/lance/pull/4316
fix: fix can not scan and take correct rowids after compaction when enable move stable rowid feature by @yanghua in https://github.com/lancedb/lance/pull/4289
fix: update broken README links by @adi-ray in https://github.com/lancedb/lance/pull/4338
fix: flaky test test_tracing.py by @ddupg in https://github.com/lancedb/lance/pull/4341
fix: correct the error enum in case of missing positions for phrase queries by @BubbleCal in https://github.com/lancedb/lance/pull/4342
fix: support count(*) by @wojiaodoubao in https://github.com/lancedb/lance/pull/4309
fix: fix bug in trim_ranges which could cause filtered read to miss rows by @westonpace in https://github.com/lancedb/lance/pull/4351
fix: process hung when using tracing events by @ddupg in https://github.com/lancedb/lance/pull/4356
fix: make sure cloud features aren't implicitly enabled (lancedb/lancedb#2567) by @vlovich in https://github.com/lancedb/lance/pull/4362
fix: implement limit and offset support for FTS and vector search by @wjones127 in https://github.com/lancedb/lance/pull/4318
fix: remove the broken large binary test with dictionary encoding && fsst compression by @HaochengLIU in https://github.com/lancedb/lance/pull/4349
fix: don't panic when reading files if the last row was deleted by @westonpace in https://github.com/lancedb/lance/pull/4396 ### Documentation 📚
docs: migrate python docs to uv based by @Xuanwo in https://github.com/lancedb/lance/pull/4346
docs: clean up table.proto by @jbapple in https://github.com/lancedb/lance/pull/4359
docs: minor fix by @lorinlee in https://github.com/lancedb/lance/pull/4379 ### Performance Improvements 🚀
perf: improve L2 distance perf by @BubbleCal in https://github.com/lancedb/lance/pull/4321
perf: pass session down when checking existing table by @wjones127 in https://github.com/lancedb/lance/pull/3677 ### Other Changes
Revert "feat: add matrix type (#4322)" by @BubbleCal in https://github.com/lancedb/lance/pull/4326
refactor: unify uri of dataset trace events by @ddupg in https://github.com/lancedb/lance/pull/4323
refactor: use previous instead v2 for our old encoding algo by @Xuanwo in https://github.com/lancedb/lance/pull/4336
refactor: simplify substrait logic, allow session state to be injected by @westonpace in https://github.com/lancedb/lance/pull/4357

New Contributors

@emmanuel-ferdman made their first contribution in https://github.com/lancedb/lance/pull/4316
@chenghao-guo made their first contribution in https://github.com/lancedb/lance/pull/4325
@adi-ray made their first contribution in https://github.com/lancedb/lance/pull/4338
@fangbo made their first contribution in https://github.com/lancedb/lance/pull/4328
@vlovich made their first contribution in https://github.com/lancedb/lance/pull/4362
@lorinlee made their first contribution in https://github.com/lancedb/lance/pull/4379

Full Changelog: https://github.com/lancedb/lance/compare/v0.32.0...v0.32.1

- Rust
Published by wjones127 7 months ago

https://github.com/lancedb/lance - v0.32.1-beta.2

What's Changed

New Features 🎉

feat: add loading trace event by @yanghua in https://github.com/lancedb/lance/pull/4298
feat: add file reader options in ReadParams by @Xuanwo in https://github.com/lancedb/lance/pull/4287
feat: add matrix type by @BubbleCal in https://github.com/lancedb/lance/pull/4322
feat(java): expose sql api for java api by @fangbo in https://github.com/lancedb/lance/pull/4328 ### Bug Fixes 🐛
fix: update missing lance perf asset by @emmanuel-ferdman in https://github.com/lancedb/lance/pull/4316
fix: fix can not scan and take correct rowids after compaction when enable move stable rowid feature by @yanghua in https://github.com/lancedb/lance/pull/4289
fix: update broken README links by @adi-ray in https://github.com/lancedb/lance/pull/4338
fix: flaky test test_tracing.py by @ddupg in https://github.com/lancedb/lance/pull/4341
fix: correct the error enum in case of missing positions for phrase queries by @BubbleCal in https://github.com/lancedb/lance/pull/4342 ### Performance Improvements 🚀
perf: improve L2 distance perf by @BubbleCal in https://github.com/lancedb/lance/pull/4321 ### Other Changes
Revert "feat: add matrix type (#4322)" by @BubbleCal in https://github.com/lancedb/lance/pull/4326
refactor: unify uri of dataset trace events by @ddupg in https://github.com/lancedb/lance/pull/4323
refactor: use previous instead v2 for our old encoding algo by @Xuanwo in https://github.com/lancedb/lance/pull/4336

New Contributors

@emmanuel-ferdman made their first contribution in https://github.com/lancedb/lance/pull/4316
@chenghao-guo made their first contribution in https://github.com/lancedb/lance/pull/4325
@adi-ray made their first contribution in https://github.com/lancedb/lance/pull/4338
@fangbo made their first contribution in https://github.com/lancedb/lance/pull/4328

Full Changelog: https://github.com/lancedb/lance/compare/v0.32.1-beta.1...v0.32.1-beta.2

- Rust
Published by wjones127 7 months ago

https://github.com/lancedb/lance - v0.32.1-beta.1

What's Changed

New Features 🎉

feat(java): brings transaction api to Java module and support project by @majin1102 in https://github.com/lancedb/lance/pull/4219 ### Bug Fixes 🐛
fix: add hnsw range search support by @LeoReeYang in https://github.com/lancedb/lance/pull/4311
fix: use debug format for object store provider errors by @jackye1995 in https://github.com/lancedb/lance/pull/4313

Full Changelog: https://github.com/lancedb/lance/compare/v0.32.0...v0.32.1-beta.1

- Rust
Published by wjones127 7 months ago

https://github.com/lancedb/lance - v0.32.0

What's Changed

Breaking Changes 🛠

feat!: use FilteredReadExec in the planner by @westonpace in https://github.com/lancedb/lance/pull/3813
feat!: consolidated index cache by @wjones127 in https://github.com/lancedb/lance/pull/4047 ### New Features 🎉
feat: allow users to control the full zip rep index cache by @Xuanwo in https://github.com/lancedb/lance/pull/4157
feat: add rle encoding support by @Xuanwo in https://github.com/lancedb/lance/pull/4177
feat: support inferring system index type by @jackye1995 in https://github.com/lancedb/lance/pull/4210
feat(rust): support trace dataset events by @yanghua in https://github.com/lancedb/lance/pull/4145
feat: support prewarm for btree by @BubbleCal in https://github.com/lancedb/lance/pull/4235
feat(java): support get lance schema with field ids by @majin1102 in https://github.com/lancedb/lance/pull/4207
feat: support sql api for dataset by @yanghua in https://github.com/lancedb/lance/pull/4086
feat: add native oss support for lance by @Xuanwo in https://github.com/lancedb/lance/pull/4247
feat: enable oss support in python and java by @Xuanwo in https://github.com/lancedb/lance/pull/4254
refactor: rewrite delete implementation to use dataset-level scanner (#4158) by @wjones127 in https://github.com/lancedb/lance/pull/4165
feat: add retry-based conflict resolution for update operations by @wjones127 in https://github.com/lancedb/lance/pull/4167
feat: add 64-bit offset support to FSST compression for large binary/string data in V2.1 by @HaochengLIU in https://github.com/lancedb/lance/pull/4092
feat: extend merge_insert fast path for UpdateIf and update-only operations by @wjones127 in https://github.com/lancedb/lance/pull/4253
feat: expose sql api for python dataset by @Jay-ju in https://github.com/lancedb/lance/pull/4267
feat(java): support replace schema and field metadata by @majin1102 in https://github.com/lancedb/lance/pull/4119
feat: add claude md for future agentic work by @Xuanwo in https://github.com/lancedb/lance/pull/4245
feat: implement byte stream split for doubles by @Xuanwo in https://github.com/lancedb/lance/pull/4236
feat: support ordering for pylance sdk by @SaintBacchus in https://github.com/lancedb/lance/pull/3556
feat: support skipping auto cleanup during write by @jackye1995 in https://github.com/lancedb/lance/pull/4293
feat: expose IVF centroids to the python API by @Sbargaoui in https://github.com/lancedb/lance/pull/4288 ### Bug Fixes 🐛
fix: forward compatibility of pylance load_indices by @jackye1995 in https://github.com/lancedb/lance/pull/4208
fix: contributing link fixed on README.md by @lalitx17 in https://github.com/lancedb/lance/pull/4211
fix: move BYTESREADCOUNTER update after await to ensure accuracy by @allenanswerzq in https://github.com/lancedb/lance/pull/4203
fix: broken image link in README by @LeoReeYang in https://github.com/lancedb/lance/pull/4228
fix: normalize_fsl should handle nulls by @Xuanwo in https://github.com/lancedb/lance/pull/4226
fix: phrase query may miss some results by @BubbleCal in https://github.com/lancedb/lance/pull/4223
fix: handle index statistics for system indexes by @jackye1995 in https://github.com/lancedb/lance/pull/4229
fix: btree may load the same page multiple times by @BubbleCal in https://github.com/lancedb/lance/pull/4246
fix: allow rowid in columns for scanner by @Jay-ju in https://github.com/lancedb/lance/pull/4116
fix: fix errors caused by simultaneous merge by @westonpace in https://github.com/lancedb/lance/pull/4279
fix: escape type keyword in tracing::info! macros by @bjurkovski in https://github.com/lancedb/lance/pull/4284
fix: proper case-insensitive comparison for filter by @wjones127 in https://github.com/lancedb/lance/pull/4278
fix: ensure _rowid comes after _distance and _score by @westonpace in https://github.com/lancedb/lance/pull/4294 ### Documentation 📚
docs: optional values are "present" or "absent" by @jbapple in https://github.com/lancedb/lance/pull/4184
docs: separate Quickstart into three tutorials by @davidmyriel in https://github.com/lancedb/lance/pull/4200
docs: add complete blob writing example and fix take_blobs usage by @TaoKevinKK in https://github.com/lancedb/lance/pull/4214
docs: add new homepage and sidebar by @davidmyriel in https://github.com/lancedb/lance/pull/4231
docs: separate table and file format spec by @jackye1995 in https://github.com/lancedb/lance/pull/4264 ### Performance Improvements 🚀
perf: improve btree performance for range query by @BubbleCal in https://github.com/lancedb/lance/pull/4248
perf: wrap Dataset with Arc in SqlQueryBuilder by @wojiaodoubao in https://github.com/lancedb/lance/pull/4275 ### Other Changes
refactor: unify two branches of add_offsets to make it more clear by @allenanswerzq in https://github.com/lancedb/lance/pull/4225
refactor: avoid fri abbrevation by @jackye1995 in https://github.com/lancedb/lance/pull/4285

New Contributors

@lalitx17 made their first contribution in https://github.com/lancedb/lance/pull/4211
@davidmyriel made their first contribution in https://github.com/lancedb/lance/pull/4200
@TaoKevinKK made their first contribution in https://github.com/lancedb/lance/pull/4214
@Sbargaoui made their first contribution in https://github.com/lancedb/lance/pull/4288

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.1...v0.32.0

- Rust
Published by wjones127 7 months ago

https://github.com/lancedb/lance - v0.32.0-beta.2

What's Changed

Breaking Changes 🛠

feat!: consolidated index cache by @wjones127 in https://github.com/lancedb/lance/pull/4047 ### New Features 🎉
feat: add claude md for future agentic work by @Xuanwo in https://github.com/lancedb/lance/pull/4245
feat: implement byte stream split for doubles by @Xuanwo in https://github.com/lancedb/lance/pull/4236 ### Bug Fixes 🐛
fix: escape type keyword in tracing::info! macros by @bjurkovski in https://github.com/lancedb/lance/pull/4284
fix: proper case-insensitive comparison for filter by @wjones127 in https://github.com/lancedb/lance/pull/4278
fix: ensure _rowid comes after _distance and _score by @westonpace in https://github.com/lancedb/lance/pull/4294 ### Other Changes
refactor: avoid fri abbrevation by @jackye1995 in https://github.com/lancedb/lance/pull/4285

Full Changelog: https://github.com/lancedb/lance/compare/v0.32.0-beta.1...v0.32.0-beta.2

- Rust
Published by wjones127 7 months ago

https://github.com/lancedb/lance - v0.32.0-beta.1

What's Changed

Breaking Changes 🛠

feat!: use FilteredReadExec in the planner by @westonpace in https://github.com/lancedb/lance/pull/3813 ### New Features 🎉
feat: support prewarm for btree by @BubbleCal in https://github.com/lancedb/lance/pull/4235
feat(java): support get lance schema with field ids by @majin1102 in https://github.com/lancedb/lance/pull/4207
feat: support sql api for dataset by @yanghua in https://github.com/lancedb/lance/pull/4086
feat: add native oss support for lance by @Xuanwo in https://github.com/lancedb/lance/pull/4247
feat: enable oss support in python and java by @Xuanwo in https://github.com/lancedb/lance/pull/4254
refactor: rewrite delete implementation to use dataset-level scanner (#4158) by @wjones127 in https://github.com/lancedb/lance/pull/4165
feat: add retry-based conflict resolution for update operations by @wjones127 in https://github.com/lancedb/lance/pull/4167
feat: add 64-bit offset support to FSST compression for large binary/string data in V2.1 by @HaochengLIU in https://github.com/lancedb/lance/pull/4092
feat: extend merge_insert fast path for UpdateIf and update-only operations by @wjones127 in https://github.com/lancedb/lance/pull/4253
feat: expose sql api for python dataset by @Jay-ju in https://github.com/lancedb/lance/pull/4267
feat(java): support replace schema and field metadata by @majin1102 in https://github.com/lancedb/lance/pull/4119 ### Bug Fixes 🐛
fix: btree may load the same page multiple times by @BubbleCal in https://github.com/lancedb/lance/pull/4246
fix: allow rowid in columns for scanner by @Jay-ju in https://github.com/lancedb/lance/pull/4116
fix: fix errors caused by simultaneous merge by @westonpace in https://github.com/lancedb/lance/pull/4279 ### Documentation 📚
docs: add complete blob writing example and fix take_blobs usage by @TaoKevinKK in https://github.com/lancedb/lance/pull/4214
docs: add new homepage and sidebar by @davidmyriel in https://github.com/lancedb/lance/pull/4231
docs: separate table and file format spec by @jackye1995 in https://github.com/lancedb/lance/pull/4264 ### Performance Improvements 🚀
perf: improve btree performance for range query by @BubbleCal in https://github.com/lancedb/lance/pull/4248
perf: wrap Dataset with Arc in SqlQueryBuilder by @wojiaodoubao in https://github.com/lancedb/lance/pull/4275 ### Other Changes
refactor: unify two branches of add_offsets to make it more clear by @allenanswerzq in https://github.com/lancedb/lance/pull/4225

New Contributors

@TaoKevinKK made their first contribution in https://github.com/lancedb/lance/pull/4214

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.2-beta.3...v0.32.0-beta.1

- Rust
Published by wjones127 7 months ago

https://github.com/lancedb/lance - v0.31.2-beta.3

What's Changed

New Features 🎉

feat(rust): support trace dataset events by @yanghua in https://github.com/lancedb/lance/pull/4145 ### Bug Fixes 🐛
fix: move BYTESREADCOUNTER update after await to ensure accuracy by @allenanswerzq in https://github.com/lancedb/lance/pull/4203
fix: broken image link in README by @LeoReeYang in https://github.com/lancedb/lance/pull/4228
fix: normalize_fsl should handle nulls by @Xuanwo in https://github.com/lancedb/lance/pull/4226
fix: phrase query may miss some results by @BubbleCal in https://github.com/lancedb/lance/pull/4223
fix: handle index statistics for system indexes by @jackye1995 in https://github.com/lancedb/lance/pull/4229 ### Documentation 📚
docs: separate Quickstart into three tutorials by @davidmyriel in https://github.com/lancedb/lance/pull/4200

New Contributors

@davidmyriel made their first contribution in https://github.com/lancedb/lance/pull/4200

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.2-beta.2...v0.31.2-beta.3

- Rust
Published by wjones127 7 months ago

https://github.com/lancedb/lance - v0.31.2-beta.2

What's Changed

New Features 🎉

feat: add rle encoding support by @Xuanwo in https://github.com/lancedb/lance/pull/4177
feat: support inferring system index type by @jackye1995 in https://github.com/lancedb/lance/pull/4210 ### Bug Fixes 🐛
fix: contributing link fixed on README.md by @lalitx17 in https://github.com/lancedb/lance/pull/4211

New Contributors

@lalitx17 made their first contribution in https://github.com/lancedb/lance/pull/4211

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.2-beta.1...v0.31.2-beta.2

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.31.2-beta.1

What's Changed

New Features 🎉

feat: allow users to control the full zip rep index cache by @Xuanwo in https://github.com/lancedb/lance/pull/4157 ### Bug Fixes 🐛
fix: forward compatibility of pylance load_indices by @jackye1995 in https://github.com/lancedb/lance/pull/4208 ### Documentation 📚
docs: optional values are "present" or "absent" by @jbapple in https://github.com/lancedb/lance/pull/4184

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.1...v0.31.2-beta.1

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.31.1

What's Changed

New Features 🎉

feat: use mkdocs for documentation by @jackye1995 in https://github.com/lancedb/lance/pull/4132
docs: add catalog integration doc section pointing to Lance Namespace by @jackye1995 in https://github.com/lancedb/lance/pull/4148
docs: add DataFusion and DuckDB documentation by @jackye1995 in https://github.com/lancedb/lance/pull/4149
feat: expose maxpagebytes arg for python LanceFileWriter by @wojiaodoubao in https://github.com/lancedb/lance/pull/4131
docs: add lancelot page by @jackye1995 in https://github.com/lancedb/lance/pull/4147
feat: add utility to capture trace events in python by @westonpace in https://github.com/lancedb/lance/pull/4123
feat: introduce MemWAL index and system index by @jackye1995 in https://github.com/lancedb/lance/pull/4069 ### Bug Fixes 🐛
fix: failed to write FTS index if the entire posting list is deleted by @BubbleCal in https://github.com/lancedb/lance/pull/4156
fix: enter span in traced object store only when polled by @wkalt in https://github.com/lancedb/lance/pull/4154
fix: shuffler fails to infer schema if the first batch is empty by @BubbleCal in https://github.com/lancedb/lance/pull/4173
fix: remapping FTS missed to remap token set by @BubbleCal in https://github.com/lancedb/lance/pull/4180
fix: update lindera and tantivy and replace lindera model by @KazuhitoT in https://github.com/lancedb/lance/pull/4144
fix: don't panic in bytewidthopt if given a fixed-size-list of a variable-width type by @westonpace in https://github.com/lancedb/lance/pull/4186 ### Documentation 📚
docs: ulimit to increase number of open files on OS X by @jbapple in https://github.com/lancedb/lance/pull/4187 ### Performance Improvements 🚀
perf: don't persist intermediate results while indexing vectors by @BubbleCal in https://github.com/lancedb/lance/pull/4004
perf: use HNSW to speed up compute_partitions by @BubbleCal in https://github.com/lancedb/lance/pull/4089
perf: use size prefixing for zstd buffer compressor for better decompressing performance by @niyue in https://github.com/lancedb/lance/pull/4029 ### Other Changes
refactor: refactor full zip schedulerangesrep by @Xuanwo in https://github.com/lancedb/lance/pull/4142
refactor: rename UploadState state change function by @allenanswerzq in https://github.com/lancedb/lance/pull/4170
refactor: implement type-safe cache interface with CacheKey trait by @wjones127 in https://github.com/lancedb/lance/pull/4075

New Contributors

@allenanswerzq made their first contribution in https://github.com/lancedb/lance/pull/4170
@jbapple made their first contribution in https://github.com/lancedb/lance/pull/4187

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.0...v0.31.1

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.31.1-beta.3

What's Changed

New Features 🎉

docs: add lancelot page by @jackye1995 in https://github.com/lancedb/lance/pull/4147 ### Bug Fixes 🐛
fix: shuffler fails to infer schema if the first batch is empty by @BubbleCal in https://github.com/lancedb/lance/pull/4173

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.1-beta.2...v0.31.1-beta.3

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.31.1-beta.2

What's Changed

Bug Fixes 🐛

fix: enter span in traced object store only when polled by @wkalt in https://github.com/lancedb/lance/pull/4154

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.1-beta.1...v0.31.1-beta.2

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.31.1-beta.1

What's Changed

New Features 🎉

feat: use mkdocs for documentation by @jackye1995 in https://github.com/lancedb/lance/pull/4132
docs: add catalog integration doc section pointing to Lance Namespace by @jackye1995 in https://github.com/lancedb/lance/pull/4148
docs: add DataFusion and DuckDB documentation by @jackye1995 in https://github.com/lancedb/lance/pull/4149
feat: expose maxpagebytes arg for python LanceFileWriter by @wojiaodoubao in https://github.com/lancedb/lance/pull/4131 ### Bug Fixes 🐛
fix: failed to write FTS index if the entire posting list is deleted by @BubbleCal in https://github.com/lancedb/lance/pull/4156 ### Performance Improvements 🚀
perf: don't persist intermediate results while indexing vectors by @BubbleCal in https://github.com/lancedb/lance/pull/4004 ### Other Changes
refactor: refactor full zip schedulerangesrep by @Xuanwo in https://github.com/lancedb/lance/pull/4142

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.0...v0.31.1-beta.1

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.31.0

What's Changed

Breaking Changes 🛠

feat!: refactor Dataset#config api and expose it via pylance by @yanghua in https://github.com/lancedb/lance/pull/4041 ### New Features 🎉
feat: add column projection to LanceFileReader by @wojiaodoubao in https://github.com/lancedb/lance/pull/4031
feat: add consecutive batch shard sampler for pytorch by @Jay-ju in https://github.com/lancedb/lance/pull/3886
feat: add Trino and PostgreSQL integrations by @jackye1995 in https://github.com/lancedb/lance/pull/4054
feat: expose num_rows api for python LanceFileReader by @yanghua in https://github.com/lancedb/lance/pull/4053
feat: add IVFHNSWFLAT index by @BubbleCal in https://github.com/lancedb/lance/pull/3671
feat: introduce IndexFileVersion for Python API by @BubbleCal in https://github.com/lancedb/lance/pull/4040
feat: add created_at for indices by @wjones127 in https://github.com/lancedb/lance/pull/4051
feat: support largestring/largebinary in lance format v2.1 by @HaochengLIU in https://github.com/lancedb/lance/pull/3967
feat: upgrade arrow object_store to 0.12.2 by @jackye1995 in https://github.com/lancedb/lance/pull/4080
feat: add random take bench based on file reader api and fragment api by @wojiaodoubao in https://github.com/lancedb/lance/pull/4076
feat: support IVF_SQ index by @BubbleCal in https://github.com/lancedb/lance/pull/4096
feat: add ignore_missing flag for safe dataset deletion by @Jay-ju in https://github.com/lancedb/lance/pull/3667
feat: add a toggle for automatic rank/world_size discovery by @Jay-ju in https://github.com/lancedb/lance/pull/3633
feat: add N-Gram tokenizer for FTS index by @BubbleCal in https://github.com/lancedb/lance/pull/4102
feat(java): support config operation for Dataset by @majin1102 in https://github.com/lancedb/lance/pull/4039
feat(datagen): add random words generator by @Xuanwo in https://github.com/lancedb/lance/pull/4099
feat: allow rowid and _rowaddr in readcolumns for dataset.addcolumns / fragment.mergecolumns by @westonpace in https://github.com/lancedb/lance/pull/4120
feat(encoding): cache repetition index for FullZip encoding by @Xuanwo in https://github.com/lancedb/lance/pull/4104 ### Bug Fixes 🐛
fix: can't search on a *_SQ index with f64 vectors by @BubbleCal in https://github.com/lancedb/lance/pull/3672
fix: torchdataset for lance does not support S3 by @Jay-ju in https://github.com/lancedb/lance/pull/4045
fix: incorrect parameter order when initializing ShardedFixedBatchSam… by @ddupg in https://github.com/lancedb/lance/pull/4064
fix: take benchmark can not read dataset by @yanghua in https://github.com/lancedb/lance/pull/4065
fix: escape type keyword in tracing::info! macros by @bjurkovski in https://github.com/lancedb/lance/pull/4068
fix: better errors on bad filters by @westonpace in https://github.com/lancedb/lance/pull/4073
fix(doc): fix doc typo in Ray integration documentation for Lance Data Sink by @kilavvy in https://github.com/lancedb/lance/pull/3984
fix: don't check vector indices when loading scalar index by @BubbleCal in https://github.com/lancedb/lance/pull/4077
fix: handle the case of all partitions are empty by @BubbleCal in https://github.com/lancedb/lance/pull/4085
fix: use std::fs::copy for local filesystem cross-filesystem copies by @wjones127 in https://github.com/lancedb/lance/pull/4088
fix: typo in feature_flags.rs by @b4l in https://github.com/lancedb/lance/pull/4093
fix: wrong cp command for java native lib name by @yanghua in https://github.com/lancedb/lance/pull/4105
fix: make maxfragmentid optional to prevent fragment ID reuse by @wjones127 in https://github.com/lancedb/lance/pull/4083
fix: panic when optimize empty PQ partition by @BubbleCal in https://github.com/lancedb/lance/pull/4108
fix: panic when commit fragment with format v2.1 by @ddupg in https://github.com/lancedb/lance/pull/4122 ### Documentation 📚
docs: fix write_dataset append mode obsolete description by @wojiaodoubao in https://github.com/lancedb/lance/pull/3866 ### Performance Improvements 🚀
perf: fast upsert with no indices by @wjones127 in https://github.com/lancedb/lance/pull/3858
perf: faster kmeans by @BubbleCal in https://github.com/lancedb/lance/pull/4070 ### Other Changes
refactor: don't use schema dictionary unless using legacy storage by @westonpace in https://github.com/lancedb/lance/pull/4023
refactor: move 2.0 code into its own module by @westonpace in https://github.com/lancedb/lance/pull/4046
refactor: move delete implementation into it's own file by @wjones127 in https://github.com/lancedb/lance/pull/4074
Revert "chore: update tantivy version to 0.24.1 (#4098)" by @Xuanwo in https://github.com/lancedb/lance/pull/4101
refactor(datagen): add RandomWord and RandomSentence to replace RandomWords by @Xuanwo in https://github.com/lancedb/lance/pull/4130

New Contributors

@wojiaodoubao made their first contribution in https://github.com/lancedb/lance/pull/4031
@ddupg made their first contribution in https://github.com/lancedb/lance/pull/4064
@bjurkovski made their first contribution in https://github.com/lancedb/lance/pull/4068
@kilavvy made their first contribution in https://github.com/lancedb/lance/pull/3984
@b4l made their first contribution in https://github.com/lancedb/lance/pull/4093
@Dig-Doug made their first contribution in https://github.com/lancedb/lance/pull/4098
@xhwhis made their first contribution in https://github.com/lancedb/lance/pull/4106

Full Changelog: https://github.com/lancedb/lance/compare/v0.30.0...v0.31.0

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.31.0-beta.2

What's Changed

New Features 🎉

feat: add a toggle for automatic rank/world_size discovery by @Jay-ju in https://github.com/lancedb/lance/pull/3633
feat: add N-Gram tokenizer for FTS index by @BubbleCal in https://github.com/lancedb/lance/pull/4102
feat(java): support config operation for Dataset by @majin1102 in https://github.com/lancedb/lance/pull/4039
feat(datagen): add random words generator by @Xuanwo in https://github.com/lancedb/lance/pull/4099 ### Bug Fixes 🐛
fix: wrong cp command for java native lib name by @yanghua in https://github.com/lancedb/lance/pull/4105
fix: make maxfragmentid optional to prevent fragment ID reuse by @wjones127 in https://github.com/lancedb/lance/pull/4083
fix: panic when optimize empty PQ partition by @BubbleCal in https://github.com/lancedb/lance/pull/4108 ### Other Changes
Revert "chore: update tantivy version to 0.24.1 (#4098)" by @Xuanwo in https://github.com/lancedb/lance/pull/4101

New Contributors

@Dig-Doug made their first contribution in https://github.com/lancedb/lance/pull/4098
@xhwhis made their first contribution in https://github.com/lancedb/lance/pull/4106

Full Changelog: https://github.com/lancedb/lance/compare/v0.31.0-beta.1...v0.31.0-beta.2

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.31.0-beta.1

What's Changed

Breaking Changes 🛠

feat!: refactor Dataset#config api and expose it via pylance by @yanghua in https://github.com/lancedb/lance/pull/4041 ### New Features 🎉
feat: add random take bench based on file reader api and fragment api by @wojiaodoubao in https://github.com/lancedb/lance/pull/4076
feat: support IVF_SQ index by @BubbleCal in https://github.com/lancedb/lance/pull/4096
feat: add ignore_missing flag for safe dataset deletion by @Jay-ju in https://github.com/lancedb/lance/pull/3667 ### Bug Fixes 🐛
fix: handle the case of all partitions are empty by @BubbleCal in https://github.com/lancedb/lance/pull/4085
fix: use std::fs::copy for local filesystem cross-filesystem copies by @wjones127 in https://github.com/lancedb/lance/pull/4088
fix: typo in feature_flags.rs by @b4l in https://github.com/lancedb/lance/pull/4093 ### Performance Improvements 🚀
perf: faster kmeans by @BubbleCal in https://github.com/lancedb/lance/pull/4070 ### Other Changes
refactor: move delete implementation into it's own file by @wjones127 in https://github.com/lancedb/lance/pull/4074

New Contributors

@b4l made their first contribution in https://github.com/lancedb/lance/pull/4093

Full Changelog: https://github.com/lancedb/lance/compare/v0.30.1-beta.1...v0.31.0-beta.1

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.30.1-beta.1

What's Changed

New Features 🎉

feat: add column projection to LanceFileReader by @wojiaodoubao in https://github.com/lancedb/lance/pull/4031
feat: add consecutive batch shard sampler for pytorch by @Jay-ju in https://github.com/lancedb/lance/pull/3886
feat: add Trino and PostgreSQL integrations by @jackye1995 in https://github.com/lancedb/lance/pull/4054
feat: expose num_rows api for python LanceFileReader by @yanghua in https://github.com/lancedb/lance/pull/4053
feat: add IVFHNSWFLAT index by @BubbleCal in https://github.com/lancedb/lance/pull/3671
feat: introduce IndexFileVersion for Python API by @BubbleCal in https://github.com/lancedb/lance/pull/4040
feat: add created_at for indices by @wjones127 in https://github.com/lancedb/lance/pull/4051
feat: support largestring/largebinary in lance format v2.1 by @HaochengLIU in https://github.com/lancedb/lance/pull/3967
feat: upgrade arrow object_store to 0.12.2 by @jackye1995 in https://github.com/lancedb/lance/pull/4080 ### Bug Fixes 🐛
fix: can't search on a *_SQ index with f64 vectors by @BubbleCal in https://github.com/lancedb/lance/pull/3672
fix: torchdataset for lance does not support S3 by @Jay-ju in https://github.com/lancedb/lance/pull/4045
fix: incorrect parameter order when initializing ShardedFixedBatchSam… by @ddupg in https://github.com/lancedb/lance/pull/4064
fix: take benchmark can not read dataset by @yanghua in https://github.com/lancedb/lance/pull/4065
fix: escape type keyword in tracing::info! macros by @bjurkovski in https://github.com/lancedb/lance/pull/4068
fix: better errors on bad filters by @westonpace in https://github.com/lancedb/lance/pull/4073
fix(doc): fix doc typo in Ray integration documentation for Lance Data Sink by @kilavvy in https://github.com/lancedb/lance/pull/3984
fix: don't check vector indices when loading scalar index by @BubbleCal in https://github.com/lancedb/lance/pull/4077 ### Documentation 📚
docs: fix write_dataset append mode obsolete description by @wojiaodoubao in https://github.com/lancedb/lance/pull/3866 ### Performance Improvements 🚀
perf: fast upsert with no indices by @wjones127 in https://github.com/lancedb/lance/pull/3858 ### Other Changes
refactor: don't use schema dictionary unless using legacy storage by @westonpace in https://github.com/lancedb/lance/pull/4023
refactor: move 2.0 code into its own module by @westonpace in https://github.com/lancedb/lance/pull/4046

New Contributors

@wojiaodoubao made their first contribution in https://github.com/lancedb/lance/pull/4031
@ddupg made their first contribution in https://github.com/lancedb/lance/pull/4064
@bjurkovski made their first contribution in https://github.com/lancedb/lance/pull/4068
@kilavvy made their first contribution in https://github.com/lancedb/lance/pull/3984

Full Changelog: https://github.com/lancedb/lance/compare/v0.30.0...v0.30.1-beta.1

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.30.0

What's Changed

Breaking Changes 🛠

feat!: auto-remap indexes before scan by @jackye1995 in https://github.com/lancedb/lance/pull/3971
feat!: move file metadata cache to bytes capacity by @wjones127 in https://github.com/lancedb/lance/pull/3949 ### New Features 🎉
feat(java): support checkout version by @yanghua in https://github.com/lancedb/lance/pull/3907
feat: wrap FilterExec in LanceFilterExec by @wkalt in https://github.com/lancedb/lance/pull/3939
feat: add storage options for LanceFileReader by @leaves12138 in https://github.com/lancedb/lance/pull/3941
feat: prefix matching for FTS by @BubbleCal in https://github.com/lancedb/lance/pull/3935
feat(java): support list versions and checkout version in Dataset by @majin1102 in https://github.com/lancedb/lance/pull/3945
feat: add a few lance examples in Rust by @HaochengLIU in https://github.com/lancedb/lance/pull/3817
feat: rebase conflict between rewrite and frag reuse index cleanup by @jackye1995 in https://github.com/lancedb/lance/pull/3952
feat(java): support tag operations on Java dataset by @majin1102 in https://github.com/lancedb/lance/pull/3958
feat(java): support restoring to the checked out version by @majin1102 in https://github.com/lancedb/lance/pull/3969
feat: expose update_config api for python by @yanghua in https://github.com/lancedb/lance/pull/3963
feat: support returning tags in order of version with descending order as default by @Jay-ju in https://github.com/lancedb/lance/pull/4001
feat: add support for must_not clause in Boolean queries by @BubbleCal in https://github.com/lancedb/lance/pull/4012
feat: support unenforced primary key concept in schema by @jackye1995 in https://github.com/lancedb/lance/pull/4002
feat: redirect Spark documentation to new lance-spark website by @jackye1995 in https://github.com/lancedb/lance/pull/4032
feat: support enable/disable auto-cleanup by @yanghua in https://github.com/lancedb/lance/pull/4037
feat: add cache stats for new cache by @wjones127 in https://github.com/lancedb/lance/pull/4024 ### Bug Fixes 🐛
fix: correctly specify schema in encoded substrait for expr by @westonpace in https://github.com/lancedb/lance/pull/3937
fix: newer indices may be read from cache by @BubbleCal in https://github.com/lancedb/lance/pull/3936
fix: null value rows missing after bitmap remap by @jackye1995 in https://github.com/lancedb/lance/pull/3956
fix: rebase transactions from oldest to latest by @jackye1995 in https://github.com/lancedb/lance/pull/3955
fix: typos in documentation comments by @leopardracer in https://github.com/lancedb/lance/pull/3970
fix: call removeoutervalidity() before dictionary_encode by @luohao in https://github.com/lancedb/lance/pull/3974
fix: don't crash when reading index trained on old version by @westonpace in https://github.com/lancedb/lance/pull/3978
fix: handle case in 2.1 writer where lists are masked by null struct by @westonpace in https://github.com/lancedb/lance/pull/3946
fix: don't panic on DF table provider if test has filter on indexed column by @westonpace in https://github.com/lancedb/lance/pull/3954
fix: respect precomputed partitions and shuffle buffers for v3 index by @BubbleCal in https://github.com/lancedb/lance/pull/3910
fix: panic caused by reading empty posting list by @BubbleCal in https://github.com/lancedb/lance/pull/4019
fix: optimize index with no new data by @wjones127 in https://github.com/lancedb/lance/pull/4017
fix: filter out null row ids by @BubbleCal in https://github.com/lancedb/lance/pull/4020
fix: separate loading configuration handling for jieba and lindera by @KazuhitoT in https://github.com/lancedb/lance/pull/3932
fix: reading large fragment reuse index details fails with file not found by @jackye1995 in https://github.com/lancedb/lance/pull/4035
fix: use i32 for dictionary encoding indices by @luohao in https://github.com/lancedb/lance/pull/3898
fix: optimizing FTS causes error if it's without positions by @BubbleCal in https://github.com/lancedb/lance/pull/4044 ### Other Changes
refactor: change the default process-wide IOPS limit to be 128 by @westonpace in https://github.com/lancedb/lance/pull/3972

New Contributors

@Xuanwo made their first contribution in https://github.com/lancedb/lance/pull/3960
@leopardracer made their first contribution in https://github.com/lancedb/lance/pull/3970

Full Changelog: https://github.com/lancedb/lance/compare/v0.29.0...v0.30.0

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.30.0-beta.1

What's Changed

Breaking Changes 🛠

feat!: auto-remap indexes before scan by @jackye1995 in https://github.com/lancedb/lance/pull/3971 ### New Features 🎉
feat: support unenforced primary key concept in schema by @jackye1995 in https://github.com/lancedb/lance/pull/4002 ### Bug Fixes 🐛
fix: optimize index with no new data by @wjones127 in https://github.com/lancedb/lance/pull/4017
fix: filter out null row ids by @BubbleCal in https://github.com/lancedb/lance/pull/4020

Full Changelog: https://github.com/lancedb/lance/compare/v0.29.1-beta.2...v0.30.0-beta.1

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.29.1-beta.2

What's Changed

New Features 🎉

feat: add storage options for LanceFileReader by @leaves12138 in https://github.com/lancedb/lance/pull/3941
feat: prefix matching for FTS by @BubbleCal in https://github.com/lancedb/lance/pull/3935
feat(java): support list versions and checkout version in Dataset by @majin1102 in https://github.com/lancedb/lance/pull/3945
feat: add a few lance examples in Rust by @HaochengLIU in https://github.com/lancedb/lance/pull/3817
feat: rebase conflict between rewrite and frag reuse index cleanup by @jackye1995 in https://github.com/lancedb/lance/pull/3952
feat(java): support tag operations on Java dataset by @majin1102 in https://github.com/lancedb/lance/pull/3958
feat(java): support restoring to the checked out version by @majin1102 in https://github.com/lancedb/lance/pull/3969
feat: expose update_config api for python by @yanghua in https://github.com/lancedb/lance/pull/3963
feat: support returning tags in order of version with descending order as default by @Jay-ju in https://github.com/lancedb/lance/pull/4001
feat: add support for must_not clause in Boolean queries by @BubbleCal in https://github.com/lancedb/lance/pull/4012 ### Bug Fixes 🐛
fix: newer indices may be read from cache by @BubbleCal in https://github.com/lancedb/lance/pull/3936
fix: null value rows missing after bitmap remap by @jackye1995 in https://github.com/lancedb/lance/pull/3956
fix: rebase transactions from oldest to latest by @jackye1995 in https://github.com/lancedb/lance/pull/3955
fix: typos in documentation comments by @leopardracer in https://github.com/lancedb/lance/pull/3970
fix: call removeoutervalidity() before dictionary_encode by @luohao in https://github.com/lancedb/lance/pull/3974
fix: don't crash when reading index trained on old version by @westonpace in https://github.com/lancedb/lance/pull/3978
fix: handle case in 2.1 writer where lists are masked by null struct by @westonpace in https://github.com/lancedb/lance/pull/3946
fix: don't panic on DF table provider if test has filter on indexed column by @westonpace in https://github.com/lancedb/lance/pull/3954
fix: respect precomputed partitions and shuffle buffers for v3 index by @BubbleCal in https://github.com/lancedb/lance/pull/3910
fix: panic caused by reading empty posting list by @BubbleCal in https://github.com/lancedb/lance/pull/4019 ### Other Changes
refactor: change the default process-wide IOPS limit to be 128 by @westonpace in https://github.com/lancedb/lance/pull/3972

New Contributors

@Xuanwo made their first contribution in https://github.com/lancedb/lance/pull/3960
@leopardracer made their first contribution in https://github.com/lancedb/lance/pull/3970

Full Changelog: https://github.com/lancedb/lance/compare/v0.29.1-beta.1...v0.29.1-beta.2

- Rust
Published by wjones127 8 months ago

https://github.com/lancedb/lance - v0.29.1-beta.1

What's Changed

New Features 🎉

feat(java): support checkout version by @yanghua in https://github.com/lancedb/lance/pull/3907
feat: wrap FilterExec in LanceFilterExec by @wkalt in https://github.com/lancedb/lance/pull/3939 ### Bug Fixes 🐛
fix: correctly specify schema in encoded substrait for expr by @westonpace in https://github.com/lancedb/lance/pull/3937

Full Changelog: https://github.com/lancedb/lance/compare/v0.29.0...v0.29.1-beta.1

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.29.0

What's Changed

Breaking Changes 🛠

perf!: improve conflict resolution performance by @wjones127 in https://github.com/lancedb/lance/pull/3882
feat!: introduce fragment reuse index to defer compaction index remap by @jackye1995 in https://github.com/lancedb/lance/pull/3847
feat!: add minimum probes and maximum probes to IVF search by @westonpace in https://github.com/lancedb/lance/pull/3903 ### New Features 🎉
feat: support slop param for phrase query by @BubbleCal in https://github.com/lancedb/lance/pull/3859
feat: enhance NGram index support for large strings by @BubbleCal in https://github.com/lancedb/lance/pull/3893
feat: support boolean query by @BubbleCal in https://github.com/lancedb/lance/pull/3879
feat: enhancing java LanceFileReader by adding projection push down and ranges lookup functionality by @leaves12138 in https://github.com/lancedb/lance/pull/3888
feat: support lance rust log in jni by @frankliee in https://github.com/lancedb/lance/pull/3901
feat: add storage options for LanceFileWriter by @leaves12138 in https://github.com/lancedb/lance/pull/3900
feat: expose auto cleanup config option for pylance by @yanghua in https://github.com/lancedb/lance/pull/3857
feat: update to the latest versions of arrow and datafusion, remove pin on chrono by @westonpace in https://github.com/lancedb/lance/pull/3890
feat: remap an index with fragment reuse index by @jackye1995 in https://github.com/lancedb/lance/pull/3906
feat: ignore indices created in newer format by @BubbleCal in https://github.com/lancedb/lance/pull/3867
feat: trim fragment reuse index after remapping by @jackye1995 in https://github.com/lancedb/lance/pull/3911
feat: add loadindexby_name to reteive a unique index by name by @jackye1995 in https://github.com/lancedb/lance/pull/3931 ### Bug Fixes 🐛
fix: remove modified fragments indices during horizontal merge_insert by @westonpace in https://github.com/lancedb/lance/pull/3863
fix: last item could be dropped from directory listing by @jackye1995 in https://github.com/lancedb/lance/pull/3892
fix: handle caching with manifest naming scheme migration by @wjones127 in https://github.com/lancedb/lance/pull/3896
fix: deserialize PhraseQuery without slop field by @wjones127 in https://github.com/lancedb/lance/pull/3913
fix: PrimitiveStructualEncoder extractvalidity buffer should follow flag keeporiginal_array by @leaves12138 in https://github.com/lancedb/lance/pull/3899
fix: unexpected posting block size when remap posting list by @BubbleCal in https://github.com/lancedb/lance/pull/3921
fix: some compile error for index by @Jay-ju in https://github.com/lancedb/lance/pull/3920
fix: skip non-leaf fields when projecting by column names in 2.1 by @westonpace in https://github.com/lancedb/lance/pull/3918
fix: prevent and handle empty fragments by @wjones127 in https://github.com/lancedb/lance/pull/3934 ### Documentation 📚
docs: fix some grammatical errors for operations comments by @majin1102 in https://github.com/lancedb/lance/pull/3930
docs: add field encoding spec by @Jay-ju in https://github.com/lancedb/lance/pull/3820
docs: add ray distributed merge columns by @Jay-ju in https://github.com/lancedb/lance/pull/3919 ### Performance Improvements 🚀
perf: remove the unnecessary copying while indexing PQ by @BubbleCal in https://github.com/lancedb/lance/pull/3894

New Contributors

@leaves12138 made their first contribution in https://github.com/lancedb/lance/pull/3888

Full Changelog: https://github.com/lancedb/lance/compare/v0.28.0...v0.29.0

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.29.0-beta.2

What's Changed

Breaking Changes 🛠

feat!: add minimum probes and maximum probes to IVF search by @westonpace in https://github.com/lancedb/lance/pull/3903 ### New Features 🎉
feat: update to the latest versions of arrow and datafusion, remove pin on chrono by @westonpace in https://github.com/lancedb/lance/pull/3890
feat: remap an index with fragment reuse index by @jackye1995 in https://github.com/lancedb/lance/pull/3906
feat: ignore indices created in newer format by @BubbleCal in https://github.com/lancedb/lance/pull/3867
feat: trim fragment reuse index after remapping by @jackye1995 in https://github.com/lancedb/lance/pull/3911
feat: add loadindexby_name to reteive a unique index by name by @jackye1995 in https://github.com/lancedb/lance/pull/3931 ### Bug Fixes 🐛
fix: deserialize PhraseQuery without slop field by @wjones127 in https://github.com/lancedb/lance/pull/3913
fix: PrimitiveStructualEncoder extractvalidity buffer should follow flag keeporiginal_array by @leaves12138 in https://github.com/lancedb/lance/pull/3899
fix: unexpected posting block size when remap posting list by @BubbleCal in https://github.com/lancedb/lance/pull/3921
fix: some compile error for index by @Jay-ju in https://github.com/lancedb/lance/pull/3920
fix: skip non-leaf fields when projecting by column names in 2.1 by @westonpace in https://github.com/lancedb/lance/pull/3918 ### Documentation 📚
docs: fix some grammatical errors for operations comments by @majin1102 in https://github.com/lancedb/lance/pull/3930
docs: add field encoding spec by @Jay-ju in https://github.com/lancedb/lance/pull/3820
docs: add ray distributed merge columns by @Jay-ju in https://github.com/lancedb/lance/pull/3919

Full Changelog: https://github.com/lancedb/lance/compare/v0.29.0-beta.1...v0.29.0-beta.2

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.29.0-beta.1

What's Changed

Breaking Changes 🛠

perf!: improve conflict resolution performance by @wjones127 in https://github.com/lancedb/lance/pull/3882
feat!: introduce fragment reuse index to defer compaction index remap by @jackye1995 in https://github.com/lancedb/lance/pull/3847 ### New Features 🎉
feat: support lance rust log in jni by @frankliee in https://github.com/lancedb/lance/pull/3901
feat: add storage options for LanceFileWriter by @leaves12138 in https://github.com/lancedb/lance/pull/3900
feat: expose auto cleanup config option for pylance by @yanghua in https://github.com/lancedb/lance/pull/3857

Full Changelog: https://github.com/lancedb/lance/compare/v0.28.1-beta.1...v0.29.0-beta.1

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.28.1-beta.1

What's Changed

New Features 🎉

feat: support slop param for phrase query by @BubbleCal in https://github.com/lancedb/lance/pull/3859
feat: enhance NGram index support for large strings by @BubbleCal in https://github.com/lancedb/lance/pull/3893
feat: support boolean query by @BubbleCal in https://github.com/lancedb/lance/pull/3879
feat: enhancing java LanceFileReader by adding projection push down and ranges lookup functionality by @leaves12138 in https://github.com/lancedb/lance/pull/3888 ### Bug Fixes 🐛
fix: remove modified fragments indices during horizontal merge_insert by @westonpace in https://github.com/lancedb/lance/pull/3863
fix: last item could be dropped from directory listing by @jackye1995 in https://github.com/lancedb/lance/pull/3892
fix: handle caching with manifest naming scheme migration by @wjones127 in https://github.com/lancedb/lance/pull/3896 ### Performance Improvements 🚀
perf: remove the unnecessary copying while indexing PQ by @BubbleCal in https://github.com/lancedb/lance/pull/3894

New Contributors

@leaves12138 made their first contribution in https://github.com/lancedb/lance/pull/3888

Full Changelog: https://github.com/lancedb/lance/compare/v0.28.0...v0.28.1-beta.1

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.28.0

What's Changed

Breaking Changes 🛠

perf!: introduce compression and new indexing algo for FTS by @BubbleCal in https://github.com/lancedb/lance/pull/3720
feat!: remove Spark module in favor of the lance-spark repo by @jackye1995 in https://github.com/lancedb/lance/pull/3700 ### New Features 🎉
feat: allow FTS and NGRAM index to co-exist on the same column by @westonpace in https://github.com/lancedb/lance/pull/3856
feat: add columns msg for count_rows by @Jay-ju in https://github.com/lancedb/lance/pull/3850
feat: auto conflict resolution for upsert by @jackye1995 in https://github.com/lancedb/lance/pull/3865
feat(java): automatically add JDK17 JVM args by @jackye1995 in https://github.com/lancedb/lance/pull/3873
feat: auto-set S3 express and fix list ordering setting by @jackye1995 in https://github.com/lancedb/lance/pull/3870 ### Bug Fixes 🐛
fix: the error that the blob field requires a project by @Jay-ju in https://github.com/lancedb/lance/pull/3745
fix: ensure to_batches output strictly adheres to configured batch size by @Jay-ju in https://github.com/lancedb/lance/pull/3808
fix: remove incorrect S3 commit documentation by @jackye1995 in https://github.com/lancedb/lance/pull/3869
fix: update lindera to 0.40.2 and correct outdated URL by @KazuhitoT in https://github.com/lancedb/lance/pull/3852
fix: pytorch demo doc format by @Jay-ju in https://github.com/lancedb/lance/pull/3860
fix: use e_tag in cache key for cached manifests by @wjones127 in https://github.com/lancedb/lance/pull/3881 ### Documentation 📚
docs: add safe dataloader for lance dataset by @Jay-ju in https://github.com/lancedb/lance/pull/3821

New Contributors

@KazuhitoT made their first contribution in https://github.com/lancedb/lance/pull/3852

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.2...v0.28.0

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.27.3-beta.2

What's Changed

New Features 🎉

feat: auto conflict resolution for upsert by @jackye1995 in https://github.com/lancedb/lance/pull/3865 ### Bug Fixes 🐛
fix: ensure to_batches output strictly adheres to configured batch size by @Jay-ju in https://github.com/lancedb/lance/pull/3808

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.3-beta.1...v0.27.3-beta.2

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.27.3-beta.1

What's Changed

New Features 🎉

feat: allow FTS and NGRAM index to co-exist on the same column by @westonpace in https://github.com/lancedb/lance/pull/3856
feat: add columns msg for count_rows by @Jay-ju in https://github.com/lancedb/lance/pull/3850 ### Bug Fixes 🐛
fix: the error that the blob field requires a project by @Jay-ju in https://github.com/lancedb/lance/pull/3745 ### Documentation 📚
docs: add safe dataloader for lance dataset by @Jay-ju in https://github.com/lancedb/lance/pull/3821

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.2...v0.27.3-beta.1

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.27.2

What's Changed

New Features 🎉

feat(java): support adding columns by schema and field definitions by @majin1102 in https://github.com/lancedb/lance/pull/3795 ### Bug Fixes 🐛
fix: update size of data file in data replacement operation by @westonpace in https://github.com/lancedb/lance/pull/3839
fix: solve cargo fmt check error under java/core/java-jni by @majin1102 in https://github.com/lancedb/lance/pull/3845
fix: allow nulls when blob data files are missing by @westonpace in https://github.com/lancedb/lance/pull/3843 ### Documentation 📚
docs: fix typos by @upczsh in https://github.com/lancedb/lance/pull/3819

New Contributors

@upczsh made their first contribution in https://github.com/lancedb/lance/pull/3819
@majin1102 made their first contribution in https://github.com/lancedb/lance/pull/3795

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.1...v0.27.2

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.27.2-beta.1

What's Changed

Bug Fixes 🐛

fix: update size of data file in data replacement operation by @westonpace in https://github.com/lancedb/lance/pull/3839 ### Documentation 📚
docs: fix typos by @upczsh in https://github.com/lancedb/lance/pull/3819

New Contributors

@upczsh made their first contribution in https://github.com/lancedb/lance/pull/3819

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.1...v0.27.2-beta.1

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.27.1

What's Changed

New Features 🎉

feat: a Datafusion FFI for the LanceTableProvider by @renato2099 in https://github.com/lancedb/lance/pull/3649 ### Bug Fixes 🐛
fix: respect prefilter transformation in ANNIvfSubIndexExec by @wkalt in https://github.com/lancedb/lance/pull/3823
fix: don't error if logging is on but tracing is off by @westonpace in https://github.com/lancedb/lance/pull/3826

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.0...v0.27.1

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.27.1-beta.1

What's Changed

Bug Fixes 🐛

fix: respect prefilter transformation in ANNIvfSubIndexExec by @wkalt in https://github.com/lancedb/lance/pull/3823

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.0...v0.27.1-beta.1

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.27.0

What's Changed

Breaking Changes 🛠

feat!: allow take_blobs API to take on row address, indices or row ids by @eddyxu in https://github.com/lancedb/lance/pull/3755 ### New Features 🎉
feat: add public accessors for lance scan fields by @wkalt in https://github.com/lancedb/lance/pull/3739
feat: store data file size in manifest by @wjones127 in https://github.com/lancedb/lance/pull/3750
feat: expose public accessor for LanceScanExec config by @wkalt in https://github.com/lancedb/lance/pull/3757
feat: fix various spans by @wjones127 in https://github.com/lancedb/lance/pull/3738
feat(build): add make clean target for Python temporary files by @Jay-ju in https://github.com/lancedb/lance/pull/3756
feat: add tracing spans to ScanExec and TakeExec by @wjones127 in https://github.com/lancedb/lance/pull/3766
feat: make it possible to filter on large_binary by @westonpace in https://github.com/lancedb/lance/pull/3797
feat: split large reads into smaller parallel reads by @westonpace in https://github.com/lancedb/lance/pull/3801
feat: introduce new FilteredReadExec which will replace the initial scan/take by @westonpace in https://github.com/lancedb/lance/pull/3782
feat: support LANCEFILEWRITERMAXPAGE_BYTES env by @yanghua in https://github.com/lancedb/lance/pull/3815 ### Bug Fixes 🐛
fix: prevent infinite loop of writing manifest by @wjones127 in https://github.com/lancedb/lance/pull/3760
fix: revert IO optimizations for now until we can test more by @wjones127 in https://github.com/lancedb/lance/pull/3763
fix: don't lose rows during bitmap remap if rows not being remapped by @westonpace in https://github.com/lancedb/lance/pull/3765
fix: flat scan frags not covered by scalar index when doing prefilter search by @westonpace in https://github.com/lancedb/lance/pull/3768
fix: partition not used yet panic by @BubbleCal in https://github.com/lancedb/lance/pull/3769
perf: reduce retry thrashing in merge_insert by @wjones127 in https://github.com/lancedb/lance/pull/3780
fix: fix mapdataset batch getitems by @Jay-ju in https://github.com/lancedb/lance/pull/3783
fix: revert tracing change from #3766 by @wjones127 in https://github.com/lancedb/lance/pull/3800
fix: limit the batch size to 64 for indexing multivectors by @BubbleCal in https://github.com/lancedb/lance/pull/3810
fix: logic bug in RowIdTreeMap union by @westonpace in https://github.com/lancedb/lance/pull/3811
fix: handle bitmap index overflow by implementing multi-batch read&write #3515 by @HaochengLIU in https://github.com/lancedb/lance/pull/3789
fix: fix python tracing by @westonpace in https://github.com/lancedb/lance/pull/3807 ### Documentation 📚
docs: how to set blob metadata by @eddyxu in https://github.com/lancedb/lance/pull/3759
docs:clarify instructions on where to run maturin develop by @HaochengLIU in https://github.com/lancedb/lance/pull/3767
docs: fix typos by @omahs in https://github.com/lancedb/lance/pull/3803 ### Performance Improvements 🚀
perf: read tiny files in 1 IOP by @wjones127 in https://github.com/lancedb/lance/pull/3752
perf: optimize load_indices by @wjones127 in https://github.com/lancedb/lance/pull/3762
perf: bring back io optimizations by @wjones127 in https://github.com/lancedb/lance/pull/3764

New Contributors

@HaochengLIU made their first contribution in https://github.com/lancedb/lance/pull/3767
@omahs made their first contribution in https://github.com/lancedb/lance/pull/3803

Full Changelog: https://github.com/lancedb/lance/compare/v0.26.1...v0.27.0

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.27.0-beta.7

What's Changed

New Features 🎉

feat: introduce new FilteredReadExec which will replace the initial scan/take by @westonpace in https://github.com/lancedb/lance/pull/3782 ### Bug Fixes 🐛
fix: limit the batch size to 64 for indexing multivectors by @BubbleCal in https://github.com/lancedb/lance/pull/3810

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.0-beta.6...v0.27.0-beta.7

- Rust
Published by wjones127 9 months ago

https://github.com/lancedb/lance - v0.27.0-beta.6

What's Changed

New Features 🎉

feat: make it possible to filter on large_binary by @westonpace in https://github.com/lancedb/lance/pull/3797
feat: split large reads into smaller parallel reads by @westonpace in https://github.com/lancedb/lance/pull/3801 ### Bug Fixes 🐛
fix: fix mapdataset batch getitems by @Jay-ju in https://github.com/lancedb/lance/pull/3783
fix: revert tracing change from #3766 by @wjones127 in https://github.com/lancedb/lance/pull/3800 ### Performance Improvements 🚀
perf: bring back io optimizations by @wjones127 in https://github.com/lancedb/lance/pull/3764

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.0-beta.5...v0.27.0-beta.6

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.27.0-beta.5

What's Changed

Bug Fixes 🐛

perf: reduce retry thrashing in merge_insert by @wjones127 in https://github.com/lancedb/lance/pull/3780 ### Performance Improvements 🚀
perf: optimize load_indices by @wjones127 in https://github.com/lancedb/lance/pull/3762

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.0-beta.4...v0.27.0-beta.5

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.27.0-beta.4

What's Changed

New Features 🎉

feat: add tracing spans to ScanExec and TakeExec by @wjones127 in https://github.com/lancedb/lance/pull/3766

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.0-beta.3...v0.27.0-beta.4

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.27.0-beta.3

What's Changed

New Features 🎉

feat: fix various spans by @wjones127 in https://github.com/lancedb/lance/pull/3738
feat(build): add make clean target for Python temporary files by @Jay-ju in https://github.com/lancedb/lance/pull/3756 ### Bug Fixes 🐛
fix: prevent infinite loop of writing manifest by @wjones127 in https://github.com/lancedb/lance/pull/3760
fix: revert IO optimizations for now until we can test more by @wjones127 in https://github.com/lancedb/lance/pull/3763
fix: don't lose rows during bitmap remap if rows not being remapped by @westonpace in https://github.com/lancedb/lance/pull/3765
fix: flat scan frags not covered by scalar index when doing prefilter search by @westonpace in https://github.com/lancedb/lance/pull/3768
fix: partition not used yet panic by @BubbleCal in https://github.com/lancedb/lance/pull/3769 ### Documentation 📚
docs: how to set blob metadata by @eddyxu in https://github.com/lancedb/lance/pull/3759
docs:clarify instructions on where to run maturin develop by @HaochengLIU in https://github.com/lancedb/lance/pull/3767 ### Performance Improvements 🚀
perf: read tiny files in 1 IOP by @wjones127 in https://github.com/lancedb/lance/pull/3752

New Contributors

@HaochengLIU made their first contribution in https://github.com/lancedb/lance/pull/3767

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.0-beta.2...v0.27.0-beta.3

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.27.0-beta.2

What's Changed

New Features 🎉

feat: expose public accessor for LanceScanExec config by @wkalt in https://github.com/lancedb/lance/pull/3757

Full Changelog: https://github.com/lancedb/lance/compare/v0.27.0-beta.1...v0.27.0-beta.2

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.27.0-beta.1

What's Changed

Breaking Changes 🛠

feat!: allow take_blobs API to take on row address, indices or row ids by @eddyxu in https://github.com/lancedb/lance/pull/3755 ### New Features 🎉
feat: store data file size in manifest by @wjones127 in https://github.com/lancedb/lance/pull/3750

Full Changelog: https://github.com/lancedb/lance/compare/v0.26.2-beta.1...v0.27.0-beta.1

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.26.2-beta.1

What's Changed

New Features 🎉

feat: add public accessors for lance scan fields by @wkalt in https://github.com/lancedb/lance/pull/3739

Full Changelog: https://github.com/lancedb/lance/compare/v0.26.1...v0.26.2-beta.1

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.26.1

What's Changed

New Features 🎉

feat: add the index params into FTS stats by @BubbleCal in https://github.com/lancedb/lance/pull/3735 ### Bug Fixes 🐛
fix: prevent deadlock using torch multi worker by @Jay-ju in https://github.com/lancedb/lance/pull/3641
ci(java): use Debian 10 with glibc 2.28 for publishing Java artifacts by @jackye1995 in https://github.com/lancedb/lance/pull/3703
fix: don't raise an error if reading a fragment with a null reader and deleted rows by @westonpace in https://github.com/lancedb/lance/pull/3741
fix: arrow version compatibility issues caused by API changes by @Jay-ju in https://github.com/lancedb/lance/pull/3748

Full Changelog: https://github.com/lancedb/lance/compare/v0.26.0...v0.26.1

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.26.1-beta.2

What's Changed

Bug Fixes 🐛

fix: don't raise an error if reading a fragment with a null reader and deleted rows by @westonpace in https://github.com/lancedb/lance/pull/3741

Full Changelog: https://github.com/lancedb/lance/compare/v0.26.1-beta.1...v0.26.1-beta.2

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.26.1-beta.1

What's Changed

New Features 🎉

feat: add the index params into FTS stats by @BubbleCal in https://github.com/lancedb/lance/pull/3735 ### Bug Fixes 🐛
fix: prevent deadlock using torch multi worker by @Jay-ju in https://github.com/lancedb/lance/pull/3641

Full Changelog: https://github.com/lancedb/lance/compare/v0.26.0...v0.26.1-beta.1

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.26.0

What's Changed

Breaking Changes 🛠

refactor(rust)!: move all object stores to use ObjectStoreProvider by @wjones127 in https://github.com/lancedb/lance/pull/3688
feat!: move object store registry to the session, re-use stores by @wjones127 in https://github.com/lancedb/lance/pull/3689 ### New Features 🎉
feat: various 2.1 fixes and performance improvements by @westonpace in https://github.com/lancedb/lance/pull/3488
feat: support to ser/deser FtsQuery by @BubbleCal in https://github.com/lancedb/lance/pull/3626
feat: add distribute add columns by ray by @Jay-ju in https://github.com/lancedb/lance/pull/3369
feat: int8 support for distance functions by @triandco in https://github.com/lancedb/lance/pull/3605
feat: support to create FTS index on list of strings by @BubbleCal in https://github.com/lancedb/lance/pull/3622
feat: simple auto cleanup by @dsgibbons in https://github.com/lancedb/lance/pull/3572
feat: support AND query and improve phrase query performance by @BubbleCal in https://github.com/lancedb/lance/pull/3623
feat: add prewarm for inverted index by @BubbleCal in https://github.com/lancedb/lance/pull/3696
feat: add a function to prewarm an index by @westonpace in https://github.com/lancedb/lance/pull/3697
feat: add bitmap (boolean) encoding to 2.1 by @westonpace in https://github.com/lancedb/lance/pull/3683
feat: add ExpLinkedList by @BubbleCal in https://github.com/lancedb/lance/pull/3721
feat: strict overwrite operation by @HubertY in https://github.com/lancedb/lance/pull/3712
feat: retry merge_insert when possible by @wjones127 in https://github.com/lancedb/lance/pull/3614 ### Bug Fixes 🐛
fix: merge insert with empty index by @wjones127 in https://github.com/lancedb/lance/pull/3624
fix: allow TorchDataset to return raw data without direct tensor conversion by @Jay-ju in https://github.com/lancedb/lance/pull/3627
fix: missing to handle FixedSizeBinary for JSON converting by @BubbleCal in https://github.com/lancedb/lance/pull/3632
fix: robust against body read failures in list endpoint by @wjones127 in https://github.com/lancedb/lance/pull/3644
fix: don't change dictionary threshold by @westonpace in https://github.com/lancedb/lance/pull/3646
fix: filter out infinite vectors by @BubbleCal in https://github.com/lancedb/lance/pull/3648
fix: check for -1 assignment during ivf-pq assignment by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3668
fix: one more finite vector fix by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3673
fix: iterabledataset constructor has no parameters by @Jay-ju in https://github.com/lancedb/lance/pull/3657
fix: disable enforcing zero copy only by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3675
fix: respect the IO parallelism when load posting lists by @BubbleCal in https://github.com/lancedb/lance/pull/3690
fix: avoid corrupting btree index when remapping nulls by @westonpace in https://github.com/lancedb/lance/pull/3704
fix: don't panic if between bounds are reversed by @westonpace in https://github.com/lancedb/lance/pull/3706
fix: the FTS deserialization is broken by @BubbleCal in https://github.com/lancedb/lance/pull/3708
fix: intersection of list of struct by @westonpace in https://github.com/lancedb/lance/pull/3665
fix: don't emit the IO core reservation warning on 2-core systems by @westonpace in https://github.com/lancedb/lance/pull/3710
fix: possible to miss some results for phrase query by @BubbleCal in https://github.com/lancedb/lance/pull/3715
fix: correctly specify datafusion properties for input distribution in our custom exec nodes by @westonpace in https://github.com/lancedb/lance/pull/3733 ### Documentation 📚
docs: add spark integration doc by @SaintBacchus in https://github.com/lancedb/lance/pull/3581 ### Performance Improvements 🚀
perf: refactor the FTS indexing algo by @BubbleCal in https://github.com/lancedb/lance/pull/3670
perf: improve ngram indexing performance by @BubbleCal in https://github.com/lancedb/lance/pull/3654 ### Other Changes
Revert "perf: refactor the FTS indexing algo (#3670)" by @BubbleCal in https://github.com/lancedb/lance/pull/3695
refactor: defer any task spanwing until first read by @westonpace in https://github.com/lancedb/lance/pull/3680

New Contributors

@triandco made their first contribution in https://github.com/lancedb/lance/pull/3605
@luohao made their first contribution in https://github.com/lancedb/lance/pull/3645
@jackye1995 made their first contribution in https://github.com/lancedb/lance/pull/3699
@HubertY made their first contribution in https://github.com/lancedb/lance/pull/3712

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.2...v0.26.0

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.26.0-beta.2

What's Changed

Breaking Changes 🛠

feat!: move object store registry to the session, re-use stores by @wjones127 in https://github.com/lancedb/lance/pull/3689 ### New Features 🎉
feat: add bitmap (boolean) encoding to 2.1 by @westonpace in https://github.com/lancedb/lance/pull/3683
feat: add ExpLinkedList by @BubbleCal in https://github.com/lancedb/lance/pull/3721
feat: strict overwrite operation by @HubertY in https://github.com/lancedb/lance/pull/3712 ### Bug Fixes 🐛
fix: avoid corrupting btree index when remapping nulls by @westonpace in https://github.com/lancedb/lance/pull/3704
fix: don't panic if between bounds are reversed by @westonpace in https://github.com/lancedb/lance/pull/3706
fix: the FTS deserialization is broken by @BubbleCal in https://github.com/lancedb/lance/pull/3708
fix: intersection of list of struct by @westonpace in https://github.com/lancedb/lance/pull/3665
fix: don't emit the IO core reservation warning on 2-core systems by @westonpace in https://github.com/lancedb/lance/pull/3710
fix: possible to miss some results for phrase query by @BubbleCal in https://github.com/lancedb/lance/pull/3715
fix: correctly specify datafusion properties for input distribution in our custom exec nodes by @westonpace in https://github.com/lancedb/lance/pull/3733

New Contributors

@jackye1995 made their first contribution in https://github.com/lancedb/lance/pull/3699
@HubertY made their first contribution in https://github.com/lancedb/lance/pull/3712

Full Changelog: https://github.com/lancedb/lance/compare/v0.26.0-beta.1...v0.26.0-beta.2

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.26.0-beta.1

What's Changed

Breaking Changes 🛠

refactor(rust)!: move all object stores to use ObjectStoreProvider by @wjones127 in https://github.com/lancedb/lance/pull/3688 ### New Features 🎉
feat: simple auto cleanup by @dsgibbons in https://github.com/lancedb/lance/pull/3572
feat: support AND query and improve phrase query performance by @BubbleCal in https://github.com/lancedb/lance/pull/3623
feat: add prewarm for inverted index by @BubbleCal in https://github.com/lancedb/lance/pull/3696
feat: add a function to prewarm an index by @westonpace in https://github.com/lancedb/lance/pull/3697 ### Bug Fixes 🐛
fix: respect the IO parallelism when load posting lists by @BubbleCal in https://github.com/lancedb/lance/pull/3690 ### Documentation 📚
docs: add spark integration doc by @SaintBacchus in https://github.com/lancedb/lance/pull/3581 ### Performance Improvements 🚀
perf: refactor the FTS indexing algo by @BubbleCal in https://github.com/lancedb/lance/pull/3670
perf: improve ngram indexing performance by @BubbleCal in https://github.com/lancedb/lance/pull/3654 ### Other Changes
Revert "perf: refactor the FTS indexing algo (#3670)" by @BubbleCal in https://github.com/lancedb/lance/pull/3695
refactor: defer any task spanwing until first read by @westonpace in https://github.com/lancedb/lance/pull/3680

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.3-beta.7...v0.26.0-beta.1

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.25.3-beta.7

What's Changed

Bug Fixes 🐛

fix: iterabledataset constructor has no parameters by @Jay-ju in https://github.com/lancedb/lance/pull/3657
fix: disable enforcing zero copy only by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3675

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.3-beta.6...v0.25.3-beta.7

- Rust
Published by wjones127 10 months ago

https://github.com/lancedb/lance - v0.25.3-beta.6

What's Changed

Bug Fixes 🐛

fix: one more finite vector fix by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3673

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.3-beta.5...v0.25.3-beta.6

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.3-beta.5

What's Changed

Bug Fixes 🐛

fix: check for -1 assignment during ivf-pq assignment by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3668

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.3-beta.4...v0.25.3-beta.5

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.3-beta.4

What's Changed

New Features 🎉

feat: support to create FTS index on list of strings by @BubbleCal in https://github.com/lancedb/lance/pull/3622 ### Bug Fixes 🐛
fix: filter out infinite vectors by @BubbleCal in https://github.com/lancedb/lance/pull/3648

New Contributors

@luohao made their first contribution in https://github.com/lancedb/lance/pull/3645

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.3-beta.3...v0.25.3-beta.4

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.3-beta.3

What's Changed

New Features 🎉

feat: int8 support for distance functions by @triandco in https://github.com/lancedb/lance/pull/3605 ### Bug Fixes 🐛
fix: robust against body read failures in list endpoint by @wjones127 in https://github.com/lancedb/lance/pull/3644
fix: don't change dictionary threshold by @westonpace in https://github.com/lancedb/lance/pull/3646

New Contributors

@triandco made their first contribution in https://github.com/lancedb/lance/pull/3605

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.3-beta.2...v0.25.3-beta.3

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.3-beta.2

What's Changed

New Features 🎉

feat: add distribute add columns by ray by @Jay-ju in https://github.com/lancedb/lance/pull/3369 ### Bug Fixes 🐛
fix: allow TorchDataset to return raw data without direct tensor conversion by @Jay-ju in https://github.com/lancedb/lance/pull/3627
fix: missing to handle FixedSizeBinary for JSON converting by @BubbleCal in https://github.com/lancedb/lance/pull/3632

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.3-beta.1...v0.25.3-beta.2

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.3-beta.1

What's Changed

New Features 🎉

feat: various 2.1 fixes and performance improvements by @westonpace in https://github.com/lancedb/lance/pull/3488
feat: support to ser/deser FtsQuery by @BubbleCal in https://github.com/lancedb/lance/pull/3626 ### Bug Fixes 🐛
fix: merge insert with empty index by @wjones127 in https://github.com/lancedb/lance/pull/3624

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.2...v0.25.3-beta.1

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.2

What's Changed

New Features 🎉

feat: add support for fixed size binary to btree by @westonpace in https://github.com/lancedb/lance/pull/3613
feat: upgrade to datafusion 46 by @wjones127 in https://github.com/lancedb/lance/pull/3618
feat: support fuzzy query and boost query by @BubbleCal in https://github.com/lancedb/lance/pull/3610 ### Bug Fixes 🐛
fix: set maximan 8 target partitions for merge insert update fragments by @LuQQiu in https://github.com/lancedb/lance/pull/3603
fix: avoid creating empty encoding task and part for PrimitiveFieldEncoder by @niyue in https://github.com/lancedb/lance/pull/3607
fix: fix python format by @Jay-ju in https://github.com/lancedb/lance/pull/3608 ### Documentation 📚
docs: add example of adding new columns with only pyarrow Field or Schema by @eddyxu in https://github.com/lancedb/lance/pull/3611
docs: add spark r/w lance demo by @yanghua in https://github.com/lancedb/lance/pull/3574 ### Performance Improvements 🚀
perf: migrate to ManifestLocation, add e_tag by @wjones127 in https://github.com/lancedb/lance/pull/3592

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.1...v0.25.2

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.2-beta.2

What's Changed

New Features 🎉

feat: upgrade to datafusion 46 by @wjones127 in https://github.com/lancedb/lance/pull/3618
feat: support fuzzy query and boost query by @BubbleCal in https://github.com/lancedb/lance/pull/3610 ### Bug Fixes 🐛
fix: fix python format by @Jay-ju in https://github.com/lancedb/lance/pull/3608 ### Documentation 📚
docs: add spark r/w lance demo by @yanghua in https://github.com/lancedb/lance/pull/3574 ### Performance Improvements 🚀
perf: migrate to ManifestLocation, add e_tag by @wjones127 in https://github.com/lancedb/lance/pull/3592

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.2-beta.1...v0.25.2-beta.2

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.2-beta.1

What's Changed

New Features 🎉

feat: add support for fixed size binary to btree by @westonpace in https://github.com/lancedb/lance/pull/3613 ### Bug Fixes 🐛
fix: set maximan 8 target partitions for merge insert update fragments by @LuQQiu in https://github.com/lancedb/lance/pull/3603
fix: avoid creating empty encoding task and part for PrimitiveFieldEncoder by @niyue in https://github.com/lancedb/lance/pull/3607 ### Documentation 📚
docs: add example of adding new columns with only pyarrow Field or Schema by @eddyxu in https://github.com/lancedb/lance/pull/3611

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.1...v0.25.2-beta.1

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.1

What's Changed

New Features 🎉

feat(python): support adding null columns with pyarrow field or schema by @eddyxu in https://github.com/lancedb/lance/pull/3602
feat: pull gcp token from env variables by @alex766 in https://github.com/lancedb/lance/pull/3583 ### Bug Fixes 🐛
fix: divide by 0 error if remapping PQ storage to empty by @BubbleCal in https://github.com/lancedb/lance/pull/3596
fix: schema isn't expected for IVF_PQ by @BubbleCal in https://github.com/lancedb/lance/pull/3606
fix: propagate parent span to spawned ObjectWriter tasks by @albertlockett in https://github.com/lancedb/lance/pull/3609 ### Performance Improvements 🚀
perf(java): cache the fragments to avoid parse the fragment json for each task by @SaintBacchus in https://github.com/lancedb/lance/pull/3599

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.0...v0.25.1

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.1-beta.3

What's Changed

Bug Fixes 🐛

fix: schema isn't expected for IVF_PQ by @BubbleCal in https://github.com/lancedb/lance/pull/3606

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.1-beta.2...v0.25.1-beta.3

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.1-beta.2

What's Changed

New Features 🎉

feat(python): support adding null columns with pyarrow field or schema by @eddyxu in https://github.com/lancedb/lance/pull/3602
feat: pull gcp token from env variables by @alex766 in https://github.com/lancedb/lance/pull/3583 ### Performance Improvements 🚀
perf(java): cache the fragments to avoid parse the fragment json for each task by @SaintBacchus in https://github.com/lancedb/lance/pull/3599

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.1-beta.1...v0.25.1-beta.2

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.1-beta.1

What's Changed

Bug Fixes 🐛

fix: divide by 0 error if remapping PQ storage to empty by @BubbleCal in https://github.com/lancedb/lance/pull/3596

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.0...v0.25.1-beta.1

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.0

What's Changed

Breaking Changes 🛠

feat!: create index in v3 version by default by @BubbleCal in https://github.com/lancedb/lance/pull/3477 ### New Features 🎉
feat: write_dataset from pylist and pydict by @eddyxu in https://github.com/lancedb/lance/pull/3527
feat: rework how we train ngram indices for better performance by @westonpace in https://github.com/lancedb/lance/pull/3518
feat: expose makedeletionsnull to python as includedeletedrows by @westonpace in https://github.com/lancedb/lance/pull/3533
feat: add project transaction operation for pylance sdk by @SaintBacchus in https://github.com/lancedb/lance/pull/3538
feat: don't log span info by @westonpace in https://github.com/lancedb/lance/pull/3547
feat: support retrain index and incremental kmeans by @BubbleCal in https://github.com/lancedb/lance/pull/3489
feat: add tracing events for I/O, index loading, and plan execution by @westonpace in https://github.com/lancedb/lance/pull/3575
feat: make it possible to get the field ids from a lance_schema by @westonpace in https://github.com/lancedb/lance/pull/3568
feat(python): add warning about fork by @wjones127 in https://github.com/lancedb/lance/pull/3584
feat: add tracing to cleanup by @wjones127 in https://github.com/lancedb/lance/pull/3585
feat: add JNI bindings for the file reader/writer by @westonpace in https://github.com/lancedb/lance/pull/3588 ### Bug Fixes 🐛
fix(java): java version is out of sync with rust and python by @yanghua in https://github.com/lancedb/lance/pull/3546
fix: indexing time in unit tests is much slower than before by @BubbleCal in https://github.com/lancedb/lance/pull/3561
fix: the PQ codes corrupted after remapping by @BubbleCal in https://github.com/lancedb/lance/pull/3573
fix: remove some expensive debug impls by @westonpace in https://github.com/lancedb/lance/pull/3576
fix: reintroduce TakeExec.dataset method by @wkalt in https://github.com/lancedb/lance/pull/3577
fix(android): compilation error on android by @TD-Sky in https://github.com/lancedb/lance/pull/3555
fix: work around deranged breaking change not labeled as such by @westonpace in https://github.com/lancedb/lance/pull/3591 ### Documentation 📚
docs: fix readandwrite example by @eddyxu in https://github.com/lancedb/lance/pull/3521
docs: organize contents into sections by @eddyxu in https://github.com/lancedb/lance/pull/3528
docs: update ray integration and move schema evolution doc to a separate doc by @eddyxu in https://github.com/lancedb/lance/pull/3530
docs: fix checklinks by @eddyxu in https://github.com/lancedb/lance/pull/3532
docs: add example of Dataset.insert by @eddyxu in https://github.com/lancedb/lance/pull/3534
docs: update README to include new table format and format v2 blogs by @eddyxu in https://github.com/lancedb/lance/pull/3535
docs: enable merge insert doctest by @eddyxu in https://github.com/lancedb/lance/pull/3542
docs: raw distributed write by @eddyxu in https://github.com/lancedb/lance/pull/3548
docs: how to use tags by @eddyxu in https://github.com/lancedb/lance/pull/3562 ### Performance Improvements 🚀
perf: coalesce continuous indices into ranges if possible by @niyue in https://github.com/lancedb/lance/pull/3513
perf: improve v3 indexing perf by @BubbleCal in https://github.com/lancedb/lance/pull/3525
perf: improve 4bit PQ performance by @BubbleCal in https://github.com/lancedb/lance/pull/3557 ### Other Changes
refactor: rework how take handles parallelism by @westonpace in https://github.com/lancedb/lance/pull/3543
refactor(rust): fix buildpredicate misleading rowids replace to row_addrs by @yanghua in https://github.com/lancedb/lance/pull/3551

New Contributors

@TD-Sky made their first contribution in https://github.com/lancedb/lance/pull/3555

Full Changelog: https://github.com/lancedb/lance/compare/v0.24.1...v0.25.0

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.0-beta.6

What's Changed

New Features 🎉

feat: make it possible to get the field ids from a lance_schema by @westonpace in https://github.com/lancedb/lance/pull/3568
feat(python): add warning about fork by @wjones127 in https://github.com/lancedb/lance/pull/3584 ### Bug Fixes 🐛
fix(android): compilation error on android by @TD-Sky in https://github.com/lancedb/lance/pull/3555 ### Performance Improvements 🚀
perf: improve 4bit PQ performance by @BubbleCal in https://github.com/lancedb/lance/pull/3557 ### Other Changes
refactor(rust): fix buildpredicate misleading rowids replace to row_addrs by @yanghua in https://github.com/lancedb/lance/pull/3551

New Contributors

@TD-Sky made their first contribution in https://github.com/lancedb/lance/pull/3555

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.0-beta.5...v0.25.0-beta.6

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.0-beta.5

What's Changed

New Features 🎉

feat: add tracing events for I/O, index loading, and plan execution by @westonpace in https://github.com/lancedb/lance/pull/3575 ### Bug Fixes 🐛
fix: the PQ codes corrupted after remapping by @BubbleCal in https://github.com/lancedb/lance/pull/3573
fix: remove some expensive debug impls by @westonpace in https://github.com/lancedb/lance/pull/3576
fix: reintroduce TakeExec.dataset method by @wkalt in https://github.com/lancedb/lance/pull/3577

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.0-beta.4...v0.25.0-beta.5

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.0-beta.4

What's Changed

New Features 🎉

feat: don't log span info by @westonpace in https://github.com/lancedb/lance/pull/3547
feat: support retrain index and incremental kmeans by @BubbleCal in https://github.com/lancedb/lance/pull/3489 ### Bug Fixes 🐛
fix(java): java version is out of sync with rust and python by @yanghua in https://github.com/lancedb/lance/pull/3546
fix: indexing time in unit tests is much slower than before by @BubbleCal in https://github.com/lancedb/lance/pull/3561 ### Documentation 📚
docs: enable merge insert doctest by @eddyxu in https://github.com/lancedb/lance/pull/3542
docs: raw distributed write by @eddyxu in https://github.com/lancedb/lance/pull/3548
docs: how to use tags by @eddyxu in https://github.com/lancedb/lance/pull/3562 ### Other Changes
refactor: rework how take handles parallelism by @westonpace in https://github.com/lancedb/lance/pull/3543

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.0-beta.3...v0.25.0-beta.4

- Rust
Published by wjones127 11 months ago

https://github.com/lancedb/lance - v0.25.0-beta.3

What's Changed

New Features 🎉

feat: add project transaction operation for pylance sdk by @SaintBacchus in https://github.com/lancedb/lance/pull/3538

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.0-beta.2...v0.25.0-beta.3

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.25.0-beta.2

What's Changed

New Features 🎉

feat: expose makedeletionsnull to python as includedeletedrows by @westonpace in https://github.com/lancedb/lance/pull/3533 ### Documentation 📚
docs: update ray integration and move schema evolution doc to a separate doc by @eddyxu in https://github.com/lancedb/lance/pull/3530
docs: fix checklinks by @eddyxu in https://github.com/lancedb/lance/pull/3532
docs: add example of Dataset.insert by @eddyxu in https://github.com/lancedb/lance/pull/3534
docs: update README to include new table format and format v2 blogs by @eddyxu in https://github.com/lancedb/lance/pull/3535 ### Performance Improvements 🚀
perf: coalesce continuous indices into ranges if possible by @niyue in https://github.com/lancedb/lance/pull/3513
perf: improve v3 indexing perf by @BubbleCal in https://github.com/lancedb/lance/pull/3525

Full Changelog: https://github.com/lancedb/lance/compare/v0.25.0-beta.1...v0.25.0-beta.2

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.25.0-beta.1

What's Changed

Breaking Changes 🛠

feat!: create index in v3 version by default by @BubbleCal in https://github.com/lancedb/lance/pull/3477 ### New Features 🎉
feat: write_dataset from pylist and pydict by @eddyxu in https://github.com/lancedb/lance/pull/3527
feat: rework how we train ngram indices for better performance by @westonpace in https://github.com/lancedb/lance/pull/3518 ### Documentation 📚
docs: fix readandwrite example by @eddyxu in https://github.com/lancedb/lance/pull/3521
docs: organize contents into sections by @eddyxu in https://github.com/lancedb/lance/pull/3528

Full Changelog: https://github.com/lancedb/lance/compare/v0.24.1...v0.25.0-beta.1

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.24.1

What's Changed

Bug Fixes 🐛

fix: the distance for multivector query is not correct by @BubbleCal in https://github.com/lancedb/lance/pull/3522

Full Changelog: https://github.com/lancedb/lance/compare/v0.24.0...v0.24.1

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.24.0

What's Changed

Breaking Changes 🛠

feat!: update DataFusion to 45.0 and Arrow to 54.1 by @timsaucer in https://github.com/lancedb/lance/pull/3503 ### New Features 🎉
feat: add support for explain analyze by @wkalt in https://github.com/lancedb/lance/pull/3484
feat(java): support delete rows from the dataset by @yanghua in https://github.com/lancedb/lance/pull/3498
feat: add support for empty structs to the 2.0 format by @westonpace in https://github.com/lancedb/lance/pull/3499
feat: cache v3 index partitions in dataset session by @BubbleCal in https://github.com/lancedb/lance/pull/3467
feat: add support for pickling fragment metadata by @westonpace in https://github.com/lancedb/lance/pull/3497
feat: support add all null column as metadata-only operation via sql by @albertlockett in https://github.com/lancedb/lance/pull/3504
feat: ConditionalPutCommitHandler for concurrency on S3, faster commit by @wjones127 in https://github.com/lancedb/lance/pull/3483
feat: add gcp token-based auth support by @alex766 in https://github.com/lancedb/lance/pull/3511
feat: emit a trace event when a significant user file is created or deleted by @westonpace in https://github.com/lancedb/lance/pull/3519 ### Bug Fixes 🐛
fix: scalar quantization can't work with NaNs by @BubbleCal in https://github.com/lancedb/lance/pull/3476
fix: ngram bench target not correct by @BubbleCal in https://github.com/lancedb/lance/pull/3490
fix: bypass the arrow take for struct array by @BubbleCal in https://github.com/lancedb/lance/pull/3500
fix: prevent despecialization of object store methods by @wjones127 in https://github.com/lancedb/lance/pull/3506
fix: the IVF/PQ centroids/codebook is with wrong data type if training on GPU by @BubbleCal in https://github.com/lancedb/lance/pull/3502
fix: pass down correct types when creating indices and items scheduler by @westonpace in https://github.com/lancedb/lance/pull/3520 ### Documentation 📚
docs: fix typo in readandwrite.rst by @ascillitoe in https://github.com/lancedb/lance/pull/3479
docs: add README.md for java module by @yanghua in https://github.com/lancedb/lance/pull/3302
docs: include create scalar index and drop index to the top level of Python API doc by @eddyxu in https://github.com/lancedb/lance/pull/3509 ### Performance Improvements 🚀
perf: parallelize ngram indexing by @BubbleCal in https://github.com/lancedb/lance/pull/3501
perf: implement XTR for retrieving multivector by @BubbleCal in https://github.com/lancedb/lance/pull/3437

New Contributors

@ascillitoe made their first contribution in https://github.com/lancedb/lance/pull/3479
@lyang24 made their first contribution in https://github.com/lancedb/lance/pull/3453
@alex766 made their first contribution in https://github.com/lancedb/lance/pull/3511
@timsaucer made their first contribution in https://github.com/lancedb/lance/pull/3503

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.2...v0.24.0

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.24.0-beta.3

What's Changed

New Features 🎉

feat: ConditionalPutCommitHandler for concurrency on S3, faster commit by @wjones127 in https://github.com/lancedb/lance/pull/3483 ### Bug Fixes 🐛
fix: the IVF/PQ centroids/codebook is with wrong data type if training on GPU by @BubbleCal in https://github.com/lancedb/lance/pull/3502 ### Documentation 📚
docs: include create scalar index and drop index to the top level of Python API doc by @eddyxu in https://github.com/lancedb/lance/pull/3509

Full Changelog: https://github.com/lancedb/lance/compare/v0.24.0-beta.2...v0.24.0-beta.3

- Rust
Published by westonpace 12 months ago

https://github.com/lancedb/lance - v0.24.0-beta.2

What's Changed

New Features 🎉

feat: cache v3 index partitions in dataset session by @BubbleCal in https://github.com/lancedb/lance/pull/3467
feat: add support for pickling fragment metadata by @westonpace in https://github.com/lancedb/lance/pull/3497
feat: support add all null column as metadata-only operation via sql by @albertlockett in https://github.com/lancedb/lance/pull/3504 ### Bug Fixes 🐛
fix: bypass the arrow take for struct array by @BubbleCal in https://github.com/lancedb/lance/pull/3500
fix: prevent despecialization of object store methods by @wjones127 in https://github.com/lancedb/lance/pull/3506 ### Performance Improvements 🚀
perf: parallelize ngram indexing by @BubbleCal in https://github.com/lancedb/lance/pull/3501
perf: implement XTR for retrieving multivector by @BubbleCal in https://github.com/lancedb/lance/pull/3437

Full Changelog: https://github.com/lancedb/lance/compare/v0.24.0-beta.1...v0.24.0-beta.2

- Rust
Published by westonpace 12 months ago

https://github.com/lancedb/lance - v0.24.0-beta.1

What's Changed

New Features 🎉

feat: add support for explain analyze by @wkalt in https://github.com/lancedb/lance/pull/3484
feat(java): support delete rows from the dataset by @yanghua in https://github.com/lancedb/lance/pull/3498
feat: add support for empty structs to the 2.0 format by @westonpace in https://github.com/lancedb/lance/pull/3499 ### Bug Fixes 🐛
fix: ngram bench target not correct by @BubbleCal in https://github.com/lancedb/lance/pull/3490 ### Documentation 📚
docs: fix typo in readandwrite.rst by @ascillitoe in https://github.com/lancedb/lance/pull/3479
docs: add README.md for java module by @yanghua in https://github.com/lancedb/lance/pull/3302

New Contributors

@ascillitoe made their first contribution in https://github.com/lancedb/lance/pull/3479
@lyang24 made their first contribution in https://github.com/lancedb/lance/pull/3453

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.3-beta.1...v0.24.0-beta.1

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.23.3-beta.1

What's Changed

Bug Fixes 🐛

fix: scalar quantization can't work with NaNs by @BubbleCal in https://github.com/lancedb/lance/pull/3476

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.2...v0.23.3-beta.1

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.23.2

What's Changed

New Features 🎉

feat: add withnewchildren implementations for several nodes by @westonpace in https://github.com/lancedb/lance/pull/3471
feat: add support for ngram indices by @westonpace in https://github.com/lancedb/lance/pull/3468 ### Bug Fixes 🐛
fix: flat FTS panic with prefilter by @BubbleCal in https://github.com/lancedb/lance/pull/3470
fix: temporarily disable spilling when training indices on string columns by @westonpace in https://github.com/lancedb/lance/pull/3469

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.1...v0.23.2

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.23.2-beta.3

What's Changed

New Features 🎉

feat: add support for ngram indices by @westonpace in https://github.com/lancedb/lance/pull/3468

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.2-beta.2...v0.23.2-beta.3

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.23.2-beta.2

What's Changed

New Features 🎉

feat: add withnewchildren implementations for several nodes by @westonpace in https://github.com/lancedb/lance/pull/3471 ### Bug Fixes 🐛
fix: temporarily disable spilling when training indices on string columns by @westonpace in https://github.com/lancedb/lance/pull/3469

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.2-beta.1...v0.23.2-beta.2

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.23.2-beta.1

What's Changed

Bug Fixes 🐛

fix: flat FTS panic with prefilter by @BubbleCal in https://github.com/lancedb/lance/pull/3470

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.1...v0.23.2-beta.1

- Rust
Published by wjones127 12 months ago

https://github.com/lancedb/lance - v0.23.1

What's Changed

New Features 🎉

feat: expose specifying scanner filters via datafusion by @westonpace in https://github.com/lancedb/lance/pull/3458
feat(python): add files lance/schema.py, lance/file.py, lance/util.py for pyright typecheck by @renato2099 in https://github.com/lancedb/lance/pull/3454
feat(java): support add columns via reader by @yanghua in https://github.com/lancedb/lance/pull/3456
feat: support to read IVF partitions by @BubbleCal in https://github.com/lancedb/lance/pull/3462 ### Bug Fixes 🐛
fix: remove extraneous padding in plain encoder by @wkalt in https://github.com/lancedb/lance/pull/3434
fix: implement withnewchildren for FTS by @BubbleCal in https://github.com/lancedb/lance/pull/3441
fix: don't eagerly materialize fields that the user hasn't asked for by @westonpace in https://github.com/lancedb/lance/pull/3442
fix: allocate much memory for residual vectors than needed by @BubbleCal in https://github.com/lancedb/lance/pull/3446
fix: flat KNN column stats order doesn't match schema by @BubbleCal in https://github.com/lancedb/lance/pull/3451 ### Performance Improvements 🚀
perf: make miniblock decoding cheaper by @westonpace in https://github.com/lancedb/lance/pull/3438

New Contributors

@wkalt made their first contribution in https://github.com/lancedb/lance/pull/3434
@renato2099 made their first contribution in https://github.com/lancedb/lance/pull/3454

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.0...v0.23.1

- Rust
Published by wjones127 about 1 year ago

https://github.com/lancedb/lance - v0.23.1-beta.5

What's Changed

New Features 🎉

feat: expose specifying scanner filters via datafusion by @westonpace in https://github.com/lancedb/lance/pull/3458

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.1-beta.4...v0.23.1-beta.5

- Rust
Published by wjones127 about 1 year ago

https://github.com/lancedb/lance - v0.23.1-beta.4

What's Changed

Bug Fixes 🐛

fix: allocate much memory for residual vectors than needed by @BubbleCal in https://github.com/lancedb/lance/pull/3446
fix: flat KNN column stats order doesn't match schema by @BubbleCal in https://github.com/lancedb/lance/pull/3451 ### Performance Improvements 🚀
perf: make miniblock decoding cheaper by @westonpace in https://github.com/lancedb/lance/pull/3438

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.1-beta.3...v0.23.1-beta.4

- Rust
Published by wjones127 about 1 year ago

https://github.com/lancedb/lance - v0.23.1-beta.3

What's Changed

Bug Fixes 🐛

fix: don't eagerly materialize fields that the user hasn't asked for by @westonpace in https://github.com/lancedb/lance/pull/3442

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.1-beta.2...v0.23.1-beta.3

- Rust
Published by wjones127 about 1 year ago

https://github.com/lancedb/lance - v0.23.1-beta.2

What's Changed

Bug Fixes 🐛

fix: implement withnewchildren for FTS by @BubbleCal in https://github.com/lancedb/lance/pull/3441

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.1-beta.1...v0.23.1-beta.2

- Rust
Published by wjones127 about 1 year ago

https://github.com/lancedb/lance - v0.23.1-beta.1

What's Changed

Bug Fixes 🐛

fix: remove extraneous padding in plain encoder by @wkalt in https://github.com/lancedb/lance/pull/3434

New Contributors

@wkalt made their first contribution in https://github.com/lancedb/lance/pull/3434

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.0...v0.23.1-beta.1

- Rust
Published by wjones127 about 1 year ago

https://github.com/lancedb/lance - v0.23.0

What's Changed

Breaking Changes 🛠

fix!: delta index fragment bitmaps contained previous index coverage by @wjones127 in https://github.com/lancedb/lance/pull/3377 ### New Features 🎉
feat: enable all datafusion functions by @westonpace in https://github.com/lancedb/lance/pull/3381
feat: support float16/float64 for multivector by @BubbleCal in https://github.com/lancedb/lance/pull/3387
feat: add drop_index by @westonpace in https://github.com/lancedb/lance/pull/3382
feat: add an all null column as a metadata-only operation by @albertlockett in https://github.com/lancedb/lance/pull/3391
feat: finish up variable-length encodings in the full-zip path by @westonpace in https://github.com/lancedb/lance/pull/3344
feat: add testing of string/binary to 2.1 full-zip encoding and fix bugs by @westonpace in https://github.com/lancedb/lance/pull/3418
feat: auto-migrate old index metadata by @wjones127 in https://github.com/lancedb/lance/pull/3428
feat: allow replacement of entire datafile when the schema lines up correctly by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3408 ### Bug Fixes 🐛
fix: json schema serializes field metadata by @albertlockett in https://github.com/lancedb/lance/pull/3379
fix: flat FTS would return all unindexed rows by @BubbleCal in https://github.com/lancedb/lance/pull/3386
fix: updating schema/field metadata now retains fragments by @albertlockett in https://github.com/lancedb/lance/pull/3384
fix: full text search index may be corrupted after remapping by @BubbleCal in https://github.com/lancedb/lance/pull/3388
fix: handle the possibility that serialize_expressions returns a memoryview by @westonpace in https://github.com/lancedb/lance/pull/3396
fix: merge_insert with subcols sometimes outputs unexpected nulls by @wjones127 in https://github.com/lancedb/lance/pull/3407
fix: ensure that 'block_size' parameter is properly propagated in the ObjectStore by @vjc578db in https://github.com/lancedb/lance/pull/3403
fix(rust): loosen bytemuck pin by @wjones127 in https://github.com/lancedb/lance/pull/3413
fix: don't compare metadata in merge insert to detect if partial schema by @westonpace in https://github.com/lancedb/lance/pull/3412
fix: support fp16 type in SQ by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3417
fix: move IO tasks off of CPU runtime in merge_insert by @wjones127 in https://github.com/lancedb/lance/pull/3420
fix: filter out null values when sampling for index training by @wjones127 in https://github.com/lancedb/lance/pull/3404
fix: handle null vectors in flat search by @wjones127 in https://github.com/lancedb/lance/pull/3422
fix: avoid divide-by-zero when training an index with a large dimension by @westonpace in https://github.com/lancedb/lance/pull/3426
fix: bump openssl for CVE by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3431 ### Performance Improvements 🚀
perf: avoid re-alloc on assigning PQ by @BubbleCal in https://github.com/lancedb/lance/pull/3399
perf: skip shuffling if there is only 1 partition by @BubbleCal in https://github.com/lancedb/lance/pull/3405

New Contributors

@vjc578db made their first contribution in https://github.com/lancedb/lance/pull/3403

Full Changelog: https://github.com/lancedb/lance/compare/v0.22.0...v0.23.0

- Rust
Published by wjones127 about 1 year ago

https://github.com/lancedb/lance - v0.23.0-beta.5

What's Changed

New Features 🎉

feat: allow replacement of entire datafile when the schema lines up correctly by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3408 ### Bug Fixes 🐛
fix: bump openssl for CVE by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3431

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.0-beta.4...v0.23.0-beta.5

- Rust
Published by wjones127 about 1 year ago

https://github.com/lancedb/lance - v0.23.0-beta.4

What's Changed

New Features 🎉

feat: auto-migrate old index metadata by @wjones127 in https://github.com/lancedb/lance/pull/3428 ### Bug Fixes 🐛
fix: avoid divide-by-zero when training an index with a large dimension by @westonpace in https://github.com/lancedb/lance/pull/3426

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.0-beta.3...v0.23.0-beta.4

- Rust
Published by wjones127 about 1 year ago

https://github.com/lancedb/lance - v0.23.0-beta.3

What's Changed

New Features 🎉

feat: finish up variable-length encodings in the full-zip path by @westonpace in https://github.com/lancedb/lance/pull/3344
feat: add testing of string/binary to 2.1 full-zip encoding and fix bugs by @westonpace in https://github.com/lancedb/lance/pull/3418 ### Bug Fixes 🐛
fix: don't compare metadata in merge insert to detect if partial schema by @westonpace in https://github.com/lancedb/lance/pull/3412
fix: support fp16 type in SQ by @chebbyChefNEQ in https://github.com/lancedb/lance/pull/3417
fix: move IO tasks off of CPU runtime in merge_insert by @wjones127 in https://github.com/lancedb/lance/pull/3420
fix: filter out null values when sampling for index training by @wjones127 in https://github.com/lancedb/lance/pull/3404
fix: handle null vectors in flat search by @wjones127 in https://github.com/lancedb/lance/pull/3422

Full Changelog: https://github.com/lancedb/lance/compare/v0.23.0-beta.2...v0.23.0-beta.3

- Rust
Published by wjones127 about 1 year ago

Recent Releases of https://github.com/lancedb/lance

https://github.com/lancedb/lance - v0.35.0-beta.2

What's Changed

New Features 🎉

https://github.com/lancedb/lance - v0.35.0-beta.1

What's Changed

New Features 🎉

https://github.com/lancedb/lance - v0.35.0

What's Changed

Breaking Changes 🛠

New Contributors

https://github.com/lancedb/lance - v0.34.0-beta.4

What's Changed

Breaking Changes 🛠

New Contributors

https://github.com/lancedb/lance - v0.34.0

What's Changed

Breaking Changes 🛠

New Contributors

https://github.com/lancedb/lance - v0.33.0-beta.3

What's Changed

New Features 🎉

https://github.com/lancedb/lance - v0.33.0-beta.2

What's Changed

New Features 🎉

https://github.com/lancedb/lance - v0.33.0-beta.1

What's Changed

New Features 🎉

https://github.com/lancedb/lance - v0.33.0

What's Changed

Breaking Changes 🛠

New Features 🎉

New Contributors

https://github.com/lancedb/lance - v0.32.2-beta.3

What's Changed

New Features 🎉

https://github.com/lancedb/lance - v0.32.2-beta.2

What's Changed

New Features 🎉

New Contributors

https://github.com/lancedb/lance - v0.32.2-beta.1

What's Changed

New Features 🎉

https://github.com/lancedb/lance - v0.32.1

What's Changed

New Features 🎉

New Contributors

https://github.com/lancedb/lance - v0.32.1-beta.2

What's Changed

New Features 🎉

New Contributors

https://github.com/lancedb/lance - v0.32.1-beta.1

What's Changed

New Features 🎉

https://github.com/lancedb/lance - v0.32.0

What's Changed

Breaking Changes 🛠

New Contributors

https://github.com/lancedb/lance - v0.32.0-beta.2

What's Changed

Breaking Changes 🛠

https://github.com/lancedb/lance - v0.32.0-beta.1

What's Changed

Breaking Changes 🛠

New Contributors

https://github.com/lancedb/lance - v0.31.2-beta.3

What's Changed

New Features 🎉

New Contributors

https://github.com/lancedb/lance - v0.31.2-beta.2

What's Changed

New Features 🎉

New Contributors

https://github.com/lancedb/lance - v0.31.2-beta.1

What's Changed

New Features 🎉

https://github.com/lancedb/lance - v0.31.1

What's Changed

New Features 🎉

New Contributors