Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of tantivy

tantivy - Tantivy v0.25

What's Changed

  • update edition to 2024 by @PSeitz in https://github.com/quickwit-oss/tantivy/pull/2620
  • make zstd optional in sstable by @Parth in https://github.com/quickwit-oss/tantivy/pull/2633
  • release tantivy: bump versions by @PSeitz in https://github.com/quickwit-oss/tantivy/pull/2625
  • Fix a typo for the comments of searchwithexecutor() by @xingtanzjr in https://github.com/quickwit-oss/tantivy/pull/2653
  • Add string fast field support to TopDocs. by @stuhood in https://github.com/quickwit-oss/tantivy/pull/2642
  • update CHANGELOG by @PSeitz in https://github.com/quickwit-oss/tantivy/pull/2621
  • fix import in test by @PSeitz-dd in https://github.com/quickwit-oss/tantivy/pull/2657
  • add docs/example and Vec values to sstable by @PSeitz in https://github.com/quickwit-oss/tantivy/pull/2660
  • fix union performance regression in tantivy 0.24 by @PSeitz-dd in https://github.com/quickwit-oss/tantivy/pull/2663
  • clippy by @PSeitz in https://github.com/quickwit-oss/tantivy/pull/2661
  • chore: Update ParadeDB logo by @philippemnoel in https://github.com/quickwit-oss/tantivy/pull/2669
  • docs: fix a typo in basic example by @DaleSeo in https://github.com/quickwit-oss/tantivy/pull/2668
  • update CHANGELOG by @Parth in https://github.com/quickwit-oss/tantivy/pull/2670
  • Fixed typo in documentation by @MassimilianoBaglioni in https://github.com/quickwit-oss/tantivy/pull/2629
  • #2635 Update fs4 to latest (0.13.1) by @1Cor125 in https://github.com/quickwit-oss/tantivy/pull/2654
  • Fix TopNComputer for reverse order by @PSeitz in https://github.com/quickwit-oss/tantivy/pull/2672
  • ignore failure to parse query when other default field suceeded by @trinity-1686a in https://github.com/quickwit-oss/tantivy/pull/2676
  • fix fieldnames in tophits aggregation by @PSeitz-dd in https://github.com/quickwit-oss/tantivy/pull/2675
  • Fix TopDocs::order_by_string_fast_field for duplicates by @stuhood in https://github.com/quickwit-oss/tantivy/pull/2673
  • feat: Support spaces between field name and value by @Darkheir in https://github.com/quickwit-oss/tantivy/pull/2678
  • test stable ordering with pagination by @PSeitz-dd in https://github.com/quickwit-oss/tantivy/pull/2683
  • per field size details by @fulmicoton in https://github.com/quickwit-oss/tantivy/pull/2679
  • convert benches to binggan by @PSeitz-dd in https://github.com/quickwit-oss/tantivy/pull/2684
  • prepare release: update Changelog by @PSeitz-dd in https://github.com/quickwit-oss/tantivy/pull/2685

New Contributors

  • @Parth made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2633
  • @xingtanzjr made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2653
  • @stuhood made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2642
  • @PSeitz-dd made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2657
  • @DaleSeo made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2668
  • @MassimilianoBaglioni made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2629
  • @1Cor125 made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2654
  • @Darkheir made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2678

Full Changelog: https://github.com/quickwit-oss/tantivy/compare/0.24...0.25.0

- Rust
Published by PSeitz 6 months ago

tantivy - Tantivy v0.24.1

Tantivy 0.24.1

  • fix: Set rust-version to 1.81

Tantivy 0.24

Tantivy 0.24 will be backwards compatible with indices created with v0.22 and v0.21. The new minimum rust version will be 1.75. Tantivy 0.23 will be skipped.

Bugfixes

  • fix potential endless loop in merge #2457(@PSeitz)
  • fix bug that causes out-of-order sstable key. #2445(@fulmicoton)
  • fix ReferenceValue API flaw #2372(@PSeitz)
  • fix OwnedBytes debug panic #2512(@b41sh)
  • catch panics during merges #2582(@rdettai)
  • switch from u32 to usize in bitpacker. This enables multivalued columns larger than 4GB, which crashed during merge before. #2581 #2586(@fulmicoton-dd @PSeitz)

Breaking API Changes

  • remove index sorting #2434(@PSeitz)

Features/Improvements

  • Aggregation
    • Support for cardinality aggregation #2337 #2446 (@raphaelcoeffic @PSeitz)
    • Support for extended stats aggregation #2247(@giovannicuccu)
    • Add Key::I64 and Key::U64 variants in aggregation to avoid f64 precision issues #2468(@PSeitz)
    • Faster term aggregation fetch terms #2447(@PSeitz)
    • Improve custom order deserialization #2451(@PSeitz)
    • Change AggregationLimits behavior #2495(@PSeitz)
    • lower contention on AggregationLimits #2394(@PSeitz)
    • fix postcard compatibility for top_hits, add postcard test #2346(@PSeitz)
    • reduce top hits memory consumption #2426(@PSeitz)
    • check unsupported parameters top_hits #2351(@PSeitz)
    • Change AggregationLimits to AggregationLimitsGuard #2495(@PSeitz)
    • add support for counting non integer in aggregation #2547(@trinity-1686a)

  • Range Queries

    • Support fast field range queries on json fields #2456(@PSeitz)
    • Add support for str fast field range query #2460 #2452 #2453(@PSeitz)
    • modify fastfield range query heuristic #2375(@trinity-1686a)
    • add FastFieldRangeQuery for explicit range queries on fast field (for RangeQuery it is autodetected) #2477(@PSeitz)

  • add format backwards-compatibility tests #2485(@PSeitz)

  • add columnar format compatibility tests #2433(@PSeitz)

  • Improved snippet ranges algorithm #2474(@gezihuzi)

  • make findfieldwith_default return json fields without path #2476(@trinity-1686a)

  • Make BooleanQuery support minimum_number_should_match #2405(@LebranceBW)

  • Make NUM_MERGE_THREADS configurable #2535(@Barre)

  • RegexPhraseQuery RegexPhraseQuery supports phrase queries with regex. E.g. query "b.* b.* wolf" matches "big bad wolf". Slop is supported as well: "b.* wolf"~2 matches "big bad wolf" #2516(@PSeitz)

  • Optional Index in Multivalue Columnar Index For mostly empty multivalued indices there was a large overhead during creation when iterating all docids (merge case). This is alleviated by placing an optional index in the multivalued index to mark documents that have values. This will slightly increase space and access time. #2439(@PSeitz)

  • Store DateTime as nanoseconds in doc store DateTime in the doc store was truncated to microseconds previously. This removes this truncation, while still keeping backwards compatibility. #2486(@PSeitz)

  • Performace/Memory

    • lift clauses in LogicalAst for optimized ast during execution #2449(@PSeitz)
    • Use Vec instead of BTreeMap to back OwnedValue object #2364(@fulmicoton)
    • Replace TantivyDocument with CompactDoc. CompactDoc is much smaller and provides similar performance. #2402(@PSeitz)
    • Recycling buffer in PrefixPhraseScorer #2443(@fulmicoton)

  • Json Type

    • JSON supports now all values on the root level. Previously an object was required. This enables support for flat mixed types. allow more JSON values, fix i64 special case #2383(@PSeitz)
    • add json path constructor to term #2367(@PSeitz)

  • QueryParser

    • fix de-escaping too much in query parser #2427(@trinity-1686a)
    • improve query parser #2416(@trinity-1686a)
    • Support field grouping title:(return AND "pink panther") #2333(@trinity-1686a)
    • allow term starting with wildcard #2568(@trinity-1686a)

  • Exist queries match subpath fields #2558(@rdettai)

  • add access benchmark for columnar #2432(@PSeitz)

  • extend indexwriter proptests #2342(@PSeitz)

  • add bench & test for columnar merging #2428(@PSeitz)

  • Change in Executor API #2391(@fulmicoton)

  • Removed usage of num_cpus #2387(@fulmicoton)

  • use bingang for agg and stacker benchmark #2378#2492(@PSeitz)

  • cleanup top level exports #2382(@PSeitz)

  • make converttofastvalueandappendtojsonterm pub #2370(@PSeitz)

  • remove JsonTermWriter #2238(@PSeitz)

  • validate sort by field type #2336(@PSeitz)

  • Fix trait bound of StoreReader::iter #2360(@adamreichold)

  • remove readpostingsno_deletes #2526(@PSeitz)

- Rust
Published by PSeitz-dd 10 months ago

tantivy - Tantivy v0.22

What's Changed

Tantivy 0.22 will be able to read indices created with Tantivy 0.21.

Bugfixes

  • Fix null byte handling in JSON paths (null bytes in json keys caused panic during indexing) #2345(@PSeitz)
  • Fix bug that can cause get_docids_for_value_range to panic. #2295(@fulmicoton)
  • Avoid 1 document indices by increase min memory to 15MB for indexing #2176(@PSeitz)
  • Fix merge panic for JSON fields #2284(@PSeitz)
  • Fix bug occuring when merging JSON object indexed with positions. #2253(@fulmicoton)
  • Fix empty DateHistogram gap bug #2183(@PSeitz)
  • Fix range query end check (fields with less than 1 value per doc are affected) #2226(@PSeitz)
  • Handle exclusive out of bounds ranges on fastfield range queries #2174(@PSeitz)

Breaking API Changes

  • rename ReloadPolicy onCommit to onCommitWithDelay #2235(@giovannicuccu)
  • Move exports from the root into modules #2220(@PSeitz)
  • Accept field name instead of Field in FilterCollector #2196(@PSeitz)
  • remove deprecated IntOptions and DateTime #2353(@PSeitz)

Features/Improvements

  • Tantivy documents as a trait: Index data directly without converting to tantivy types first #2071(@ChillFish8)
  • encode some part of posting list as -1 instead of direct values (smaller inverted indices) #2185(@trinity-1686a)

  • Aggregation

    • Support to deserialize f64 from string #2311(@PSeitz)
    • Add a top_hits aggregator #2198(@ditsuke)
    • Support bool type in term aggregation #2318(@PSeitz)
    • Support ip adresses in term aggregation #2319(@PSeitz)
    • Support date type in term aggregation #2172(@PSeitz)
    • Support escaped dot when addressing field #2250(@PSeitz)

  • Add ExistsQuery to check documents that have a value #2160(@imotov)

  • Expose TopDocs::orderbyu64_field again #2282(@ditsuke)

  • Memory/Performance

    • Faster TopN: replace BinaryHeap with TopNComputer #2186(@PSeitz)
    • reduce number of allocations during indexing #2257(@PSeitz)
    • Less Memory while indexing: docid deltas while indexing #2249(@PSeitz)
    • Faster indexing: use term hashmap in fastfield #2243(@PSeitz)
    • term hashmap remove copy in isempty, unused unorderedid #2229(@PSeitz)
    • add method to fetch block of first values in columnar #2330(@PSeitz)
    • Faster aggregations: add fast path for full columns in fetch_block #2328(@PSeitz)
    • Faster sstable loading: use fst for sstable index #2268(@trinity-1686a)

  • QueryParser

    • allow newline where we allow space in query parser #2302(@trinity-1686a)
    • allow some mixing of occur and bool in strict query parser #2323(@trinity-1686a)
    • handle * inside term in lenient query parser #2228(@trinity-1686a)
    • add support for exists query syntax in query parser #2170(@trinity-1686a)

  • Add shared search executor #2312(@MochiXu)

  • Truncate keys to u16::MAX in term hashmap #2299(@PSeitz)

  • report if a term matched when warming up posting list #2309(@trinity-1686a)

  • Support json fields in FuzzyTermQuery #2173(@PingXia-at)

  • Read list of fields encoded in term dictionary for JSON fields #2184(@PSeitz)

  • add collect_block to BoxableSegmentCollector #2331(@PSeitz)

  • expose collect_block buffer size #2326(@PSeitz)

  • Forward regex parser errors #2288(@adamreichold)

  • Make FacetCounts defaultable and cloneable. #2322(@adamreichold)

  • Derive Debug for SchemaBuilder #2254(@GodTamIt)

  • add missing inlines to tantivy options #2245(@PSeitz)

New Contributors

  • @imotov made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2160
  • @PingXia-at made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2173
  • @giovannicuccu made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2235
  • @BlackHoleFox made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2265
  • @ditsuke made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2282
  • @MochiXu made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2312

Full Changelog: https://github.com/quickwit-oss/tantivy/compare/0.21...v0.22.0

- Rust
Published by PSeitz almost 2 years ago

tantivy - Tantivy v0.21

Bugfixes

  • Fix track fast field memory consumption, which led to higher memory consumption than the budget allowed during indexing #2148#2147(@PSeitz)
  • Fix a regression from 0.20 where sort index by date wasn't working anymore #2124(@PSeitz)
  • Fix getting the root facet on the FacetCollector. #2086(@adamreichold)
  • Align numerical type priority order of columnar and query. #2088(@fmassot) ### Breaking Changes
  • Remove support for Brotli and Snappy compression #2123(@adamreichold) ### Features/Improvements
  • Implement lenient query parser #2129(@trinity-1686a)
  • orderbyu64field and orderbyfastfield allow sorting in ascending and descending order #2111(@naveenann)
  • Allow dynamic filters in text analyzer builder #2110(@fulmicoton @fmassot)
  • Aggregation
    • Add missing parameter for term aggregation #2149#2103(@PSeitz)
    • Add missing parameter for percentiles #2157(@PSeitz)
    • Add missing parameter for stats,min,max,count,sum,avg #2151(@PSeitz)
    • Improve aggregation deserialization error message #2150(@PSeitz)
    • Add validation for type Bytes to term_agg #2077(@PSeitz)
    • Alternative mixed field collection #2135(@PSeitz)
  • Add missing query_terms impl for TermSetQuery. #2120(@adamreichold)
  • Minor improvements to OwnedBytes #2134(@adamreichold)
  • Remove allocations in split compound words #2080(@PSeitz)
  • Ngram tokenizer now returns an error with invalid arguments #2102(@fmassot)
  • Make TextAnalyzerBuilder public #2097(@adamreichold)
  • Return an error when tokenizer is not found while indexing #2093(@naveenann)
  • Delayed column opening during merge #2132(@PSeitz)

New Contributors

  • @naveenann made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2093
  • @cjrh made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2144
  • @GodTamIt made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2145
  • @ethever made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2165

Full Changelog: https://github.com/quickwit-oss/tantivy/compare/0.20.2...0.21

- Rust
Published by PSeitz over 2 years ago

tantivy - 0.20.1

What's Changed

  • Fix building on windows with mmap by @ChillFish8 in https://github.com/quickwit-oss/tantivy/pull/2070

- Rust
Published by PSeitz over 2 years ago

tantivy - Tantivy v0.20

What's Changed

Bugfixes

  • Fix phrase queries with slop (slop supports now transpositions, algorithm that carries slop so far for num terms > 2) #2031#2020(@PSeitz)
  • Handle error for exists on MMapDirectory #1988 (@PSeitz)
  • Aggregation
    • Fix min doc_count empty merge bug #2057 (@PSeitz)
    • Fix: Sort order for term aggregations (sort order on key was inverted) #1858 (@PSeitz)

Features/Improvements

  • Add PhrasePrefixQuery #1842 (@trinity-1686a)
  • Add coerce option for text and numbers types (convert the value instead of returning an error during indexing) #1904 (@PSeitz)
  • Add regex tokenizer #1759(@mkleen)
  • Move tokenizer API to seperate crate. Having a seperate crate with a stable API will allow us to use tokenizers with different tantivy versions. #1767 (@PSeitz)
  • Columnar crate: New fast field handling (@fulmicoton @PSeitz) #1806#1809
    • Support for fast fields with optional values. Previously tantivy supported only single-valued and multi-value fast fields. The encoding of optional fast fields is now very compact.
    • Fast field Support for JSON (schemaless fast fields). Support multiple types on the same column. #1876 (@fulmicoton)
    • Unified access for fast fields over different cardinalities.
    • Unified storage for typed and untyped fields.
    • Move fastfield codecs into columnar. #1782 (@fulmicoton)
    • Sparse dense index for optional values #1716 (@PSeitz)
    • Switch to nanosecond precision in DateTime fastfield #2016 (@PSeitz)
  • Aggregation
    • Add date_histogram aggregation (only fixed_interval for now) #1900 (@PSeitz)
    • Add percentiles aggregations #1984 (@PSeitz)
    • [breaking] Drop JSON support on intermediate agg result (we use postcard as format in quickwit to send intermediate results) #1992 (@PSeitz)
    • Set memory limit in bytes for aggregations after which they abort (Previously there was only the bucket limit) #1942#1957(@PSeitz)
    • Add support for u64,i64,f64 fields in term aggregation #1883 (@PSeitz)
    • Add count, min, max, and sum aggregations #1794 (@guilload)
    • Switch to Aggregation without serde_untagged => better deserialization errors. #2003 (@PSeitz)
    • Switch to ms in histogram for date type (ES compatibility) #2045 (@PSeitz)
    • Reduce term aggregation memory consumption #2013 (@PSeitz)
    • Reduce agg memory consumption: Replace generic aggregation collector (which has a high memory requirement per instance) in aggregation tree with optimized versions behind a trait.
    • Split term collection count and sub_agg (Faster term agg with less memory consumption for cases without sub-aggs) #1921 (@PSeitz)
    • Schemaless aggregations: In combination with stacker tantivy supports now schemaless aggregations via the JSON type.
    • Add aggregation support for JSON type #1888 (@PSeitz)
    • Mixed types support on JSON fields in aggs #1971 (@PSeitz)
    • Perf: Fetch blocks of vals in aggregation for all cardinality #1950 (@PSeitz)
  • Searcher with disabled scoring via EnableScoring::Disabled #1780 (@shikhar)
  • Enable tokenizer on json fields #2053 (@PSeitz)
  • Enforcing "NOT" and "-" queries consistency in UserInputAst #1609 (@Denis Bazhenov)
  • Faster indexing
    • Refactor tokenization pipeline to use GATs #1924 (@trinity-1686a)
    • Faster term hash map #1940 (@PSeitz)
    • Refactor vint #2010 (@PSeitz)
  • Faster search
    • Work in batches of docs on the SegmentCollector (Only for cases without score for now) #1937 (@PSeitz)
    • Faster fast field range queries using SIMD #1954 (@fulmicoton)
    • Improve fast field range query performance #1864 (@PSeitz)
  • Make BM25 scoring more flexible #1855 (@alexcole)
  • Switch fs2 to fs4 as it is now unmaintained and does not support illumos #1944 (@Toasterson)
  • Made BooleanWeight and BoostWeight public #1991 (@fulmicoton)
  • Make index compatible with virtual drives on Windows #1843 (@Yukun Guo)
  • Auto downgrade index record option, instead of vint error #1857 (@PSeitz)
  • Enable range query on fast field for u64 compatible types #1762 (@PSeitz) [#1876]
  • sstable
    • Isolating sstable and stacker in independant crates. #1718 (@fulmicoton)
    • New sstable format #1943#1953 (@trinity-1686a)
    • Use DeltaReader directly to implement Dictionnary::ordtoterm #1928 (@trinity-1686a)
    • Use DeltaReader directly to implement Dictionnary::term_ord #1925 (@trinity-1686a)
  • Add seperate tokenizer manager for fast fields #2019 (@PSeitz)
  • Make construction of LevenshteinAutomatonBuilder for FuzzyTermQuery instances lazy. #1756 (@adamreichold)
  • Added support for madvise when opening an mmaped Index #2036 (@fulmicoton)
  • Rename DatePrecision to DateTimePrecision #2051 (@guilload)
  • Query Parser
    • Quotation mark can now be used for phrase queries. #2050 (@fulmicoton)
    • PhrasePrefixQuery is supported in the query parser via: field:"phrase ter"* #2044 (@adamreichold)
  • Docs
    • Update examples for literate docs #1880 (@PSeitz)
    • Add ip field example #1775 (@PSeitz)
    • Fix doc store cache documentation #1821 (@PSeitz)
    • Fix BooleanQuery document #1999 (@RT_Enzyme)
    • Update comments in the faceted search example #1737 (@DawChihLiou)

New Contributors

  • @mhlakhani made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1733
  • @pinkforest made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1746
  • @DawChihLiou made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1737
  • @mkleen made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1759
  • @lonre made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1803
  • @gyk made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1843
  • @alexcole made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1855
  • @Toasterson made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1944
  • @vsop-479 made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1970
  • @Tony-X made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1985
  • @RTEnzyme made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1999
  • @tottoto made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2018
  • @nyurik made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2038
  • @bazhenov made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1609
  • @lavrd made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1422
  • @tnxbutno made their first contribution in https://github.com/quickwit-oss/tantivy/pull/2069

Full Changelog: https://github.com/quickwit-oss/tantivy/blob/main/CHANGELOG.md

- Rust
Published by PSeitz over 2 years ago

tantivy - Tantivy v0.19.2

Fixes an issue in the skip list deserialization, which deserialized the byte start offset incorrectly as u32. get_doc will fail for any docs that live in a block with start offset larger than u32::MAX (~4GB). Causes index corruption, if a segment with a doc store file larger 4GB is merged. (@PSeitz)

- Rust
Published by PSeitz about 3 years ago

tantivy - Tantivy v0.19.1

Hotfix on handling user input for getdocidforvaluerange (@PSeitz)

- Rust
Published by PSeitz about 3 years ago

tantivy - Tantivy v0.19

What's Changed

Bugfixes

  • Fix missing fieldnorms for u64, i64, f64, bool, bytes and date #1620 (@PSeitz)
  • Fix interpolation overflow in linear interpolation fastfield codec #1480 (@PSeitz @fulmicoton)

Features/Improvements

  • Add support for IN in queryparser , e.g. field: IN [val1 val2 val3] #1683 (@trinity-1686a)
  • Skip score calculation, when no scoring is required #1646 (@PSeitz)
  • Limit fast fields to u32 (get_val(u32)) #1644 (@PSeitz)
  • The DateTime type has been updated to hold timestamps with microseconds precision. DateOptions and DatePrecision have been added to configure Date fields. The precision is used to hint on fast values compression. Otherwise, seconds precision is used everywhere else (i.e terms, indexing) #1396 (@evanxg852000)
  • Add IP address field type #1553 (@PSeitz)
  • Add boolean field type #1382 (@boraarslan)
  • Remove Searcher pool and make Searcher cloneable. (@PSeitz)
  • Validate settings on create #1570 (@PSeitz)
  • Detect and apply gcd on fastfield codecs #1418 (@PSeitz)
  • Doc store
    • use separate thread to compress block store #1389 #1510 (@PSeitz @fulmicoton)
    • Expose doc store cache size #1403 (@PSeitz)
    • Enable compression levels for doc store #1378 (@PSeitz)
    • Make block size configurable #1374 (@kryesh)
  • Make tantivy::TantivyError cloneable #1402 (@PSeitz)
  • Add support for phrase slop in query language #1393 (@saroh)
  • Aggregation
    • Add aggregation support for date type #1693(@PSeitz)
    • Add support for keyed parameter in range and histgram aggregations #1424 (@k-yomo)
    • Add aggregation bucket limit #1363 (@PSeitz)
  • Faster indexing
    • #1610 (@PSeitz)
    • #1594 (@PSeitz)
    • #1582 (@PSeitz)
    • #1611 (@PSeitz)
    • Added a pre-configured stop word filter for various language #1666 (@adamreichold)

New Contributors

  • @ryanrussell made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1380
  • @boraarslan made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1382
  • @pier-oliviert made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1415
  • @kianmeng made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1445
  • @akr4 made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1473
  • @waywardmonkeys made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1524
  • @nigel-andrews made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1608
  • @theduke made their first contribution in https://github.com/quickwit-oss/tantivy/pull/1624

Full Changelog: https://github.com/quickwit-oss/tantivy/compare/0.18...0.19

- Rust
Published by PSeitz about 3 years ago

tantivy - Tantivy v0.18.1

  • Hotfix on positions. #1629 (@fmassot, @fulmicoton, @PSeitz)

- Rust
Published by fulmicoton over 3 years ago

tantivy - Tantivy v0.18

  • For date values chrono has been replaced with time (@uklotzde) #1304
  • Add histgram aggregation (@PSeitz)
  • Add support for fastfield on text fields (@PSeitz)
  • Add terms aggregation (@PSeitz)
  • Add support for zstd compression (@kryesh)

- Rust
Published by fulmicoton over 3 years ago

tantivy - Tantivy v0.17

  • LogMergePolicy now triggers merges if the ratio of deleted documents reaches a threshold (@shikhar @fulmicoton) #115
  • Adds a searcher Warmer API (@shikhar @fulmicoton)
  • Change to non-strict schema. Ignore fields in data which are not defined in schema. Previously this returned an error. #1211
  • Facets are necessarily indexed. Existing index with indexed facets should work out of the box. Index without facets that are marked with index: false should be broken (but they were already broken in a sense). (@fulmicoton) #1195 .
  • Bugfix that could in theory impact durability in theory on some filesystems #1224
  • Schema now offers not indexing fieldnorms (@lpouget) #922
  • Reduce the number of fsync calls #1225
  • Fix opening bytes index with dynamic codec (@PSeitz) #1278
  • Added an aggregation collector compatible with Elasticsearch (@PSeitz)
  • Added a JSON schema type @fulmicoton #1251
  • Added support for slop in phrase queries @halvorboe #1068

- Rust
Published by fulmicoton almost 4 years ago

tantivy - Tantivy v0.16.1

Major Bugfix on multivalued fastfield. #1151

- Rust
Published by fulmicoton over 4 years ago

tantivy - Tantivy 0.15.3

  • Major bugfix. Deleting documents was broken when the index was sorted by a field. (@appaquet, @fulmicoton) #1101

- Rust
Published by fulmicoton over 4 years ago

tantivy - Tantivy 0.15.2

  • Major bugfix. DocStore still panics when a deleted doc is at the beginning of a block. (@appaquet) #1088

- Rust
Published by fulmicoton over 4 years ago

tantivy - Tantivy 0.15.1

  • Major bugfix. DocStore panics when first block is deleted. (@appaquet) #1077

- Rust
Published by fulmicoton over 4 years ago

tantivy - Tantivy 0.15

  • API Changes. Using Range instead of (start, end) in the API and internals (FileSlice, OwnedBytes, Snippets, ...) This change is breaking but migration is trivial.
  • Added an Histogram collector. (@fulmicoton) #994
  • Added support for Option. (@fulmicoton)
  • DocAddress is now a struct (@scampi) #987
  • Bugfix consistent tie break handling in facet's topk (@hardikpnsp) #357
  • Date field support for range queries (@rihardsk) #516
  • Added lz4-flex as the default compression scheme in tantivy (@PSeitz) #1009
  • Renamed a lot of symbols to avoid all uppercasing on acronyms, as per new clippy recommendation. For instance, RAMDirectory -> RamDirectory. (@fulmicoton)
  • Simplified positions index format (@fulmicoton) #1022
  • Moved bitpacking to bitpacker subcrate and add BlockedBitpacker, which bitpacks blocks of 128 elements (@PSeitz) #1030
  • Added support for more-like-this query in tantivy (@evanxg852000) #1011
  • Added support for sorting an index, e.g presorting documents in an index by a timestamp field. This can heavily improve performance for certain scenarios, by utilizing the sorted data (Top-n optimizations)(@PSeitz). #1026
  • Add iterator over documents in doc store (@PSeitz). #1044
  • Fix log merge policy (@PSeitz). #1043
  • Add detection to avoid small doc store blocks on merge (@PSeitz). #1054
  • Make doc store compression dynamic (@PSeitz). #1060
  • Switch to json for footer version handling (@PSeitz). #1060
  • Updated TermMerger implementation to rely on the union feature of the FST (@scampi) #469
  • Add boolean marking whether position is required in the query_terms API call (@fulmicoton). #1070

- Rust
Published by fulmicoton over 4 years ago

tantivy - Tantivy 0.14

  • Remove dependency to atomicwrites #833 .Implemented by @fulmicoton upon suggestion and research from @asafigan).
  • Migrated tantivy error from the now deprecated failure crate to thiserror #760. (@hirevo)
  • API Change. Accessing the typed value off a Schema::Value now returns an Option instead of panicking if the type does not match.
  • Large API Change in the Directory API. Tantivy used to assume that all files could be somehow memory mapped. After this change, Directory return a FileSlice that can be reduced and eventually read into an OwnedBytes object. Long and blocking io operation are still required by they do not span over the entire file.
  • Added support for Brotli compression in the DocStore. (@ppodolsky)
  • Added helper for building intersections and unions in BooleanQuery (@guilload)
  • Bugfix in Query::explain
  • Removed dependency on notify #924. Replaced with FileWatcher struct that polls meta file every 500ms in background thread. (@halvorboe @guilload)
  • Added FilterCollector, which wraps another collector and filters docs using a predicate over a fast field (@barrotsteindev)
  • Simplified the encoding of the skip reader struct. BlockWAND max tf is now encoded over a single byte. (@fulmicoton)
  • FilterCollector now supports all Fast Field value types (@barrotsteindev)
  • FastField are not all loaded when opening the segment reader. (@fulmicoton)

This version breaks compatibility and requires users to reindex everything.

- Rust
Published by fulmicoton about 5 years ago

tantivy - Tantivy 0.13.3

Minor Bugfix. Avoid relying on serde's reexport of PhantomData. (#975)

- Rust
Published by fulmicoton about 5 years ago

tantivy - Tantivy 0.13.2

HotFix. Acquiring a facet reader on a segment that does not contain any doc with this facet returns None. (#896)

- Rust
Published by fulmicoton over 5 years ago

tantivy - Tantivy 0.13.1

Made Query and Collector Send + Sync. Updated misc dependency versions.

- Rust
Published by fulmicoton over 5 years ago

tantivy - Tantivy 0.13

Tantivy 0.13 introduce a change in the index format that will require you to reindex your index (BlockWAND information are added in the skiplist). The index size increase is minor as this information is only added for full blocks. If you have a massive index for which reindexing is not an option, please contact me so that we can discuss possible solutions.

  • Bugfix in FuzzyTermQuery not matching terms by prefix when it should (@Peachball)
  • Relaxed constraints on the custom/tweak score functions. At the segment level, they can be mut, and they are not required to be Sync + Send.
  • MMapDirectory::open does not return a Result anymore.
  • Change in the DocSet and Scorer API. (@fulmicoton). A freshly created DocSet point directly to their first doc. A sentinel value called TERMINATED marks the end of a DocSet. .advance() returns the new DocId. Scorer::skip(target) has been replaced by Scorer::seek(target) and returns the resulting DocId. As a result, iterating through DocSet now looks as follows rust let mut doc = docset.doc(); while doc != TERMINATED { // ... doc = docset.advance(); } The change made it possible to greatly simplify a lot of the docset's code.
  • Misc internal optimization and introduction of the Scorer::for_each_pruning function. (@fulmicoton)
  • Added an offset option to the Top(.*)Collectors. (@robyoung)
  • Added Block WAND. Performance on TOP-K on term-unions should be greatly increased. (@fulmicoton, and special thanks to the PISA team for answering all my questions!)

- Rust
Published by fulmicoton over 5 years ago

tantivy - Tantivy 0.12

  • Removing static dispatch in tokenizers for simplicity. (#762)
  • Added backward iteration for TermDictionary stream. (@halvorboe)
  • Fixed a performance issue when searching for the posting lists of a missing term (@audunhalland)
  • Added a configurable maximum number of docs (10M by default) for a segment to be considered for merge (@hntd187, landed by @halvorboe #713)
  • Important Bugfix #777, causing tantivy to retain memory mapping. (diagnosed by @poljar)
  • Added support for field boosting. (#547, @fulmicoton)

- Rust
Published by fulmicoton about 6 years ago

tantivy - Tantivy 0.11.3

  • Fixed DateTime as a fast field (#735)

- Rust
Published by fulmicoton about 6 years ago

tantivy - Tantivy 0.11.1

  • Bug fix #729

- Rust
Published by fulmicoton about 6 years ago

tantivy - Tantivy 0.11.0

  • Added f64 field. Internally reuse u64 code the same way i64 does (@fdb-hiroshima)
  • Various bugfixes in the query parser.
    • Better handling of hyphens in query parser. (#609)
    • Better handling of whitespaces.
  • Closes #498 - add support for Elastic-style unbounded range queries for alphanumeric types eg. "title:>hello", "weight:>=70.5", "height:<200" (@petr-tik)
  • API change around Box<BoxableTokenizer>. See detail in #629
  • Avoid rebuilding Regex automaton whenever a regex query is reused. #639 (@brainlock)
  • Add footer with some metadata to index files. #605 (@fdb-hiroshima)
  • Add a method to check the compatibility of the footer in the index with the running version of tantivy (@petr-tik)
  • TopDocs collector: ensure stable sorting on equal score. #671 (@brainlock)
  • Added handling of pre-tokenized text fields (#642), which will enable users to load tokens created outside tantivy. See usage in examples/pretokenizedtext. (@kkoziara)
  • Fix crash when committing multiple times with deleted documents. #681 (@brainlock)

How to update?

  • The index format is changed. You are required to reindex your data to use tantivy 0.11.
  • Box<dyn BoxableTokenizer> has been replaced by a BoxedTokenizer struct.
  • Regex are now compiled when the RegexQuery instance is built. As a result, it can now return an error and handling the Result is required.
  • tantivy::version() now returns a Version object. This object implements ToString()

- Rust
Published by fulmicoton about 6 years ago

tantivy - Tantivy 0.10.3

  • Fix crash when committing multiple times with deleted documents. #681 (@brainlock)

- Rust
Published by fulmicoton over 6 years ago

tantivy - Tantivy 0.10.2

Hotfix for #656

- Rust
Published by fulmicoton over 6 years ago

tantivy - Tantivy 0.10.1

  • Closes #544. A few users experienced problems with the directory watching system. Avoid watching the mmap directory until someone effectively creates a reader that uses this functionality.

- Rust
Published by fulmicoton over 6 years ago

tantivy - Tantivy 0.10.0

Tantivy 0.10.0 index format is compatible with the index format in 0.9.0.

  • Added an API to easily tweak or entirely replace the default score. See TopDocs::tweak_scoreand TopScore::custom_score (@pmasurel)
  • Added an ASCII folding filter (@drusellers)
  • Bugfix in query.count in presence of deletes (@pmasurel)
  • Added .explain(...) in Query and Weight to (@pmasurel)
  • Added an efficient way to delete_all_documents in IndexWriter (@petr-tik). All segments are simply removed.

Minor

  • Switched to Rust 2018 (@uvd)
  • Small simplification of the code. Calling .freq() or .doc() when .advance() has never been called on segment postings should panic from now on.
  • Tokens exceeding u16::max_value() - 4 chars are discarded silently instead of panicking.
  • Fast fields are now preloaded when the SegmentReader is created.
  • IndexMeta is now public. (@hntd187)
  • IndexWriter add_document, delete_term. IndexWriter is Sync, making it possible to use it with a Arc<RwLock<IndexWriter>>. add_document and delete_term can only require a read lock. (@pmasurel)
  • Introducing Opstamp as an expressive type alias for u64. (@petr-tik)
  • Stamper now relies on AtomicU64 on all platforms (@petr-tik)
  • Bugfix - Files get deleted slightly earlier
  • Compilation resources improved (@fdb-hiroshima)

How to update?

Your program should be usable as is.

Fast fields

Fast fields used to be accessed directly from the SegmentReader. The API changed, you are now required to acquire your fast field reader via the segment_reader.fast_fields(), and use one of the typed method: - .u64(), .i64() if your field is single-valued ; - .u64s(), .i64s() if your field is multi-valued ; - .bytes() if your field is bytes fast field.

- Rust
Published by fulmicoton over 6 years ago

tantivy - Tantivy 0.9.1

Hotfix . All language were using the English stemmer.

- Rust
Published by fulmicoton almost 7 years ago

tantivy - Tantivy 0.9

0.9.0 index format is not compatible with the previous index format.

Bugfix

Some Mmap objects were being leaked, and would never get released. (@fulmicoton)

New Features

  • Added IndexReader. By default, index is reloaded automatically upon new commits (@fulmicoton)
  • Stemming in other language possible (@pentlander)
  • Added grouped add and delete operations. They are guaranteed to happen together (i.e. they cannot be split by a commit). In addition, adds are guaranteed to happen on the same segment. (@elbow-jason)
  • Added DateTime field (@barrotsteindev)

Misc improvements

  • Indexer memory footprint improved. (VInt comp, inlining the first block. (@fulmicoton)
  • Removed most unsafe (@fulmicoton)
  • Segments with no docs are deleted earlier (@barrotsteindev)
  • Removed INT_STORED and INT_INDEXED. It is now possible to use STORED and INDEXED for int fields. (@fulmicoton)

- Rust
Published by fulmicoton almost 7 years ago

tantivy - Tantivy 0.8.2

0.8.2 fixes build for non x86_64 platforms. See #496 for details.

- Rust
Published by fulmicoton about 7 years ago

tantivy - Tantivy 0.8.1

Hotfix of #476.

Merge was reflecting deletes before commit was passed. Thanks @barrotsteindev for reporting the bug.

- Rust
Published by fulmicoton about 7 years ago

tantivy - Tantivy 0.8.0

  • API Breaking change in the collector API. (@jwolfe, @fulmicoton)
  • Multithreaded search (@jwolfe, @fulmicoton)

- Rust
Published by fulmicoton about 7 years ago

tantivy - Tantivy 0.7.2

Bugfix #457 Removing faulty debug_assert!.

- Rust
Published by fulmicoton about 7 years ago

tantivy - Tantivy 0.7.1

  • Bugfix: NGramTokenizer panics on non ascii chars
  • Added a space usage API

- Rust
Published by fulmicoton over 7 years ago

tantivy - Tantivy 0.7.0

  • Skip data for doc ids and positions (@fulmicoton), greatly improving performance
  • Tantivy error now rely on the failure crate (@drusellers)
  • Added support for AND, OR, NOT syntax in addition to the +,- syntax
  • Added a snippet generator with highlight (@vigneshsarma, @fulmicoton)
  • Added a TopFieldCollector (@pentlander)

- Rust
Published by fulmicoton over 7 years ago

tantivy - Tantivy 0.6.1

  • Bugfix #324. GC removing was removing file that were still in u seful
  • Added support for parsing AllQuery and RangeQuery via QueryParser
    • AllQuery: *
    • RangeQuery:
      • Inclusive field:[startIncl to endIncl]
      • Exclusive field:{startExcl to endExcl}
      • Mixed field:[startIncl to endExcl} and vice versa
      • Unbounded field:[start to *], field:[* to end]

- Rust
Published by fulmicoton over 7 years ago

tantivy - Tantivy 0.6.0

Special thanks to @drusellers and @jason-wolfe for their contributions to this release!

From now on Tantivy compiles on stable rust.

  • Removed C code. Tantivy is now pure Rust. (@pmasurel)
  • BM25 (@pmasurel)
  • Approximate field norms encoded over 1 byte. (@pmasurel)
  • Compiles on stable rust (@pmasurel)
  • Add &[u8] fastfield for associating arbitrary bytes to each document (@jason-wolfe) (#270)
    • Completely uncompressed
    • Internally: One u64 fast field for indexes, one fast field for the bytes themselves.
  • Add NGram token support (@drusellers)
  • Add Stopword Filter support (@drusellers)
  • Add a FuzzyTermQuery (@drusellers)
  • Add a RegexQuery (@drusellers)
  • Various performance improvements (@pmasurel)_

- Rust
Published by fulmicoton over 7 years ago

tantivy - Tantivy 0.5.2

Hotfix of 0.5.x for the following issues - bugfix #274 - bugfix #280 - bugfix #289

- Rust
Published by fulmicoton almost 8 years ago

tantivy - Tantivy 0.5.1

Bugfix #254 : tantivy failed if no documents in a segment contained a specific field.

- Rust
Published by fulmicoton almost 8 years ago

tantivy - Tantivy 0.5.0

  • Faceting
  • RangeQuery
  • Configurable tokenization pipeline
  • Bugfix in PhraseQuery
  • Various query optimisation
  • Allowing very large indexes
    • 64 bits file address
    • Smarter encoding of the TermInfo objects

- Rust
Published by fulmicoton about 8 years ago

tantivy - Tantivy 0.4.2

  • No more AVX in the release binaries.
  • HOTFIX: Bugfix for unindexed fields.

- Rust
Published by fulmicoton over 8 years ago

tantivy - 0.4.0 Faster indexing

  • Raise the limit of number of fields (previously 256 fields) (@fulmicoton)
  • Removed u32 fields. They are replaced by u64 and i64 fields (#65) (@fulmicoton)
  • Optimized skip in SegmentPostings (#130) (@lnicola)
  • Replacing rustc_serialize by serde. Kudos to @KodrAus and @lnicola
  • Using error-chain (@KodrAus)
  • QueryParser: (@fulmicoton)
    • Explicit error returned when searched for a term that is not indexed
    • Searching for a int term via the query parser was broken (age:1)
    • Searching for a non-indexed field returns an explicit Error
    • Phrase query for non-tokenized field are not tokenized by the query parser.
  • Faster/Better indexing (@fulmicoton)
    • using murmurhash2
    • faster merging
    • more memory efficient fast field writer (@lnicola )
    • better handling of collisions
    • lesser memory usage
  • Added API, most notably to iterate over ranges of terms (@fulmicoton)
  • Bugfix that was preventing to unmap segment files, on index drop (@fulmicoton)
  • Made the doc! macro public (@fulmicoton)
  • Added an alternative implementation of the streaming dictionary (@fulmicoton)

- Rust
Published by fulmicoton over 8 years ago

tantivy - HOTFIX Added a garbage_collect_files method

Added a garbagecollectfiles method to IndexWriter

- Rust
Published by fulmicoton almost 9 years ago

tantivy - Tantivy 0.3: Delete and Windows support

Special thanks to @Kodraus @lnicola @Ameobea @manuel-woelker @celaus for their contribution to this release.

Thanks also to everyone in tantivy gitter chat for their advise and company :)

https://gitter.im/tantivy-search/tantivy

Warning

Tantivy 0.3 is NOT backward compatible with tantivy 0.2 code and index format. You should not expect backward compatibility before tantivy 1.0.

New Features

  • Delete. You can now delete documents from an index.
  • Support for windows (Thanks to @lnicola)

Various Bugfixes & small improvements

  • Added CI for Windows (https://ci.appveyor.com/project/fulmicoton/tantivy) Thanks to @KodrAus ! (#108)
  • Various dependy version update (Thanks to @Ameobea) #76
  • Fixed several race conditions in Index.wait_merge_threads
  • Fixed #72. Mmap were never released.
  • Fixed #80. Fast field used to take an amplitude of 32 bits after a merge. (Ouch!)
  • Fixed #92. u32 are now encoded using big endian in the fst in order to make there enumeration consistent with the natural ordering.
  • Building binary targets for tantivy-cli (Thanks to @KodrAus)
  • Misc invisible bug fixes, and code cleanup.

- Rust
Published by fulmicoton almost 9 years ago

tantivy -

- Rust
Published by fulmicoton about 9 years ago