vearch - v3.5.7

| Vearch Version | Python SDK Version | Go SDK Version | |----------------|--------------------|----------------| | v3.5.7 | v3.5.4 | v3.5.3 |

What's Changed

Added * feat: Add incremental backup support for RocksDB storage * feat: rebuild index * feat: Add cluster health promethues metric * feat: support "=" filter and "OR" filter operator * feat: Add zstd compression for backup files and implement decompressi… * feat: add region parameter to S3 configuration for backup service *

Changed * ci: split cluster ci * refactor: Move the compilation of faiss to env to improve CI speed * refactor: Update metric handling in profiler and monitor service * chore: update faiss version to v1.10.0

Fixed * fix: update objectName format to include clusterName in backup handler * fix: add nil checks for monitoring system initialization in Profiler … * fix: restore from checkpoint snapshot instead of raw data * fix: destory column family handle before close database

New Contributors

• @yanwr1 made their first contribution in https://github.com/vearch/vearch/pull/833

Full Changelog: https://github.com/vearch/vearch/compare/v3.5.6...v3.5.7

- Go
Published by zcdb 10 months ago

vearch - v3.5.6

| Vearch Version | Python SDK Version | Go SDK Version | |----------------|--------------------|----------------| | v3.5.6 | v3.5.4 | v3.5.3 |

What's Changed

Added * feat: add option for get document by hash

Changed * refactor: split master services into separate files * feat: implement Query method in MultiFieldsRangeIndex for optimized document retrieval * refactor: update metrics handling to use pointer types and improve protobuf generation * perf: change space replica num default value to 3 * perf: reduce bitmap copies * chore(deps): bump golang.org/x/net in /tools/backup

Fixed * fix: check duplicate id when get or upsert * fix: set table size as maxdocid and check updating in engine * fix: check partition num with partition rule * fix: query from leader when update * fix: add error handling for GetVector method in VectorManager * fix: check space field name * fix(python sdk): set space replica_num * fix: optimize RawVectors access in Engine and VectorManager to fix stack crash

Full Changelog: https://github.com/vearch/vearch/compare/v3.5.5...v3.5.6

- Go
Published by zcdb 12 months ago

vearch - v3.5.5

| Vearch Version | Python SDK Version | Go SDK Version | |----------------|--------------------|----------------| | v3.5.5 | v3.5.3 | v3.5.3 |

What's Changed

Added * chore: add "spaceconfig" in master; update dependencies for bytedance/sonic and google.golang.org/grpc by @wxingda in https://github.com/vearch/vearch/pull/816 * Add langchain4j in readme and example by @Martin7-1 in https://github.com/vearch/vearch/pull/817 * feat: add interface for replace master member and auto sync router an… by @zcdb in https://github.com/vearch/vearch/pull/819 * ci: check master members in cicluster.yml by @zcdb in https://github.com/vearch/vearch/pull/825 * feat(python sdk): support filter operator of NOT IN and new data type * feat: add longsearchtime config to record slow search request * feat: add router registration logic in NewServer function * feat: add interface for replace master member and auto sync router an… * feat: automatically recover on other healthy ps server when replica fail * feat: add replica anti-affinity strategy * feat: add request ID handling and space scanning in server logic; upd…

Changed * ci: enable automatic recovery of failed nodes in configuration * refactor: enhance storage manager with new Get and MultiGet methods, … * feat: string array data type can set as string_array * feat: update data types to int64_t for improved precision and add ToR… * refactor: use timeout midware instead of timeout handling in document…

Fixed * fix: 1. fix multi-vector search memory leak; 2. remove VIDMgr; * fix: add timeout handling and retry logic in RegisterRouter method * fix: check ivfpq index score is valid * fix: check upsert result with duplicate id * fix: add minimum training threshold * fix: fix dead lock in range search and crash in delete scalar index * fix: check cluster health * fix: use topn when merging results to avoid empty results for some shards * fix: fix string array index length limit

New Contributors

• @Martin7-1 made their first contribution in https://github.com/vearch/vearch/pull/817

Full Changelog: https://github.com/vearch/vearch/compare/v3.5.4...v3.5.5

- Go
Published by wxingda about 1 year ago

vearch - v3.5.5

What's Changed

Added * chore: add "spaceconfig" in master; update dependencies for bytedance/sonic and google.golang.org/grpc by @wxingda in https://github.com/vearch/vearch/pull/816 * Add langchain4j in readme and example by @Martin7-1 in https://github.com/vearch/vearch/pull/817 * feat: add interface for replace master member and auto sync router an… by @zcdb in https://github.com/vearch/vearch/pull/819 * ci: check master members in cicluster.yml by @zcdb in https://github.com/vearch/vearch/pull/825 * feat(python sdk): support filter operator of NOT IN and new data type * feat: add longsearchtime config to record slow search request * feat: add router registration logic in NewServer function * feat: add interface for replace master member and auto sync router an… * feat: automatically recover on other healthy ps server when replica fail * feat: add replica anti-affinity strategy * feat: add request ID handling and space scanning in server logic; upd…

Changed * ci: enable automatic recovery of failed nodes in configuration * refactor: enhance storage manager with new Get and MultiGet methods, … * feat: string array data type can set as string_array * feat: update data types to int64_t for improved precision and add ToR… * refactor: use timeout midware instead of timeout handling in document…

Fixed * fix: 1. fix multi-vector search memory leak; 2. remove VIDMgr; * fix: add timeout handling and retry logic in RegisterRouter method * fix: check ivfpq index score is valid * fix: check upsert result with duplicate id * fix: add minimum training threshold * fix: fix dead lock in range search and crash in delete scalar index * fix: check cluster health

New Contributors

• @Martin7-1 made their first contribution in https://github.com/vearch/vearch/pull/817

Full Changelog: https://github.com/vearch/vearch/compare/v3.5.4...v3.5.5

- Go
Published by wxingda about 1 year ago

vearch - v3.5.4

| Vearch Version | Python SDK Version | Go SDK Version | |----------------|--------------------|----------------| | v3.5.4 | v3.5.3 | v3.5.3 |

What's Changed

Added * chore: simplify image creation process using docker multi-stage builds * feat: set request_id in document search interface

Removed * refactor: reduce one JSON serialization operation in the return result and remove unnecessary proto fields

Changed * refactor: Refactor search result merging and sorting logic to reduce memory and CPU usage * refactor: unify the engine serialization method of search responses to protobuffers * refactor: unify python sdk * test: set token for python sdk test * refactor: use static library for rocksdb in build script * refactor: optimize load fieldrangeindex_ addition in multi-threaded environment * perf: use fieldrwlocks_ locking instead of rwlock to improve scalar indexing performance.

Fixed * refactor: optimize response handling by removing intermediate tmpItems variable * fix: python sdk can get space detail info and set search parameters when create space * fix: truncate doc num on loading to prevent raft duplicate insertion * fix: 1. fix put /fail/server/ to etcd failure; 2. fix /schedule/recover_server remove failed server from space meta data * fix: check the index parameter of the vector field of space

New Contributors

• @kaleaditya779 made their first contribution in https://github.com/vearch/vearch/pull/812

Full Changelog: https://github.com/vearch/vearch/compare/v3.5.3...v3.5.4

- Go
Published by wxingda over 1 year ago

vearch - v3.5.3

| Vearch Version | Python SDK Version | Go SDK Version | |----------------|--------------------|----------------| | v3.5.3 | v3.5.2 | v3.5.3 |

What's Changed

Added * ci: add arm64 image build * ci: add error param test case in backup * feat: dockerfile use sleep instead of vearch * refactor: add EQ and LIKE as todo in python sdk * ci: add cluster index * feat: add term filter NOT IN condition

Removed * refactor: remove malloc_trim in engine.cc

Changed * refactor: use ShouldBindJSON instead of vjson.Unmarshal * ci: update docker-arch comparition * ci: use stop and start instead of restart in ci_cluster * ci: use stream9-minimal instead of stream9 to reduce image size * Update DeployByDockerZH_CN.md, fix vearch-dev-env url error

Fixed * ci:fix backup test case error * fix: check delete result whether has error * fix: hnsw index read and write thread competition * fix: check docid when GetDocByDocID * fix: hnsw index read and write thread competition * Update index.py: Fix NamedTuple Immutability Issue in IvfPQIndex and … * ix: change index parameter buckertmaxsize * fix: fix numeric scalar index search crash when reach the last key * fix: fix replicas faulty num and add test cases * fix: return error information when getSpace has error

New Contributors

• @cococo2000 made their first contribution in https://github.com/vearch/vearch/pull/798

Full Changelog: https://github.com/vearch/vearch/compare/v3.5.2...v3.5.3

- Go
Published by wxingda over 1 year ago

vearch - v3.5.2

Added

• feat: custom sharding for range sharding
• feat: add data type of Date
• feat: rbac authentication
• feat: implement modify engine config
• feat: backup with schema
• feat: benchmark scripts
• feat: resource limit
• feat: add backupSpace interface
• feat: adjust codes to compatible arm64

Removed

• chore: remove unused code and update file paths in table_io.h

Changed

• build(deps): bump github.com/gin-contrib/cors from 1.5.0 to 1.6.0
• build: bump protoc from v3.6.1 to v3.21.0
• ci: bump up image to centos:stream9
• refactor: separate query from search in engine
• refactor: move EngineStatus to entity
• build: update rocksdb version to v9.2.1
• refactor: update some default value in python sdk
• refactor: update query documents
• remove text field index in langchain sdk
• refactor: use json instead of flatbuffers in memory_info
• refactor: use json instead of flatbuffers in engine_status
• chore: Refactor WeightedRanker to use nlohmann/json for parsing
• refactor: use std::vector instead of a character array to avoid stack memory allocation issues
• refactor: using threadskv8 as the sole B+ tree

Fixed

• fix: create root user failed then check again
• fix: fix backup with multi partitions
• fix: fix partition id error for server and change partition member error
• fix: proxy retry all master node until success
• fix fieldrangeindex thread synchronization bug
• fix: add table_->GetFieldRawValue return value check
• fix: python sdk fix auth bug
• fix: keep float32 field precision

Full Changelog: https://github.com/vearch/vearch/compare/v3.5.1...v3.5.2

- Go
Published by zcdb over 1 year ago

vearch - v3.5.1

Added

• add go sdk
• add python sdk
• add openapi
• add trace option for document interface

Removed

• Remove some unnecessary logic in document query

Changed

• alloc FieldOperate in stack to reduce new/delete
• merge io operator to raw_vector
• use StorageManager in RawVector
• chore: use unconditional strings.TrimSuffix
• Using reference counting instead of deferred release

Fixed

• check index MetricType is empty
• delete data directionary after engine cloesed
• wait for engine closure before deleting the data directory

New Contributors

• @testwill made their first contribution in https://github.com/vearch/vearch/pull/783

Full Changelog: https://github.com/vearch/vearch/compare/v3.5.0...v3.5.1

- Go
Published by zcdb almost 2 years ago

vearch - v3.5.0

Added

• add rocksdb bitmap manager
• add engine search and createtable error msg in response
• add data type stringArray
• ivfflat support brute force search
• Add status to manage function return values
• add alias interface ### Removed
• remove useless error code
• remove boost in search request
• remove OnlineLogLevel
• remove MSearch
• remove response field _source
• remove document vector field feature in query response
• remove support for search condition of document_ids
• remove upsert vector field.feature
• remove result field.extra and table.compress_mode
• remove support for indivisible nsubvector of index ivfpq
• remove id_type ### Changed
• unify error return format
• reduce memory copy in table Serialize
• return status in CreateVectorTable
• separate query from search method
• Classify the HTTP return values of Vearch as standard HTTP status codes
• use recall_num instead of quick
• rename search size to limit
• reformat query/filters
• Unify return data format of document interface
• Unify return data format of cluster version interface
• Unify the format of meta define interface
• rename indexsize to trainingthreshold
• set default value for index_params to simplify space schema
• update space operation interface
• deal with log error without directly exit ### Fixed
• call free only when an error occurs
• fix request auth

- Go
Published by zcdb almost 2 years ago

vearch - v3.4.4

Added

• support query next undeleted document with a docid value
• add retry for listen pprof port
• add interface of space/describe
• add vearch_backup tool
• add new version python sdk struct ### Removed
• remove retrieval_types ### Changed
• switch the router http framework to gin
• simplify error messages
• change FieldType in space to vearchpb.FieldType
• change some error return for document interface
• replace cbjson builder by sonic
• replace ioutil by os ### Fixed
• fix deleteByQuery nil result panic
• fix GetMemoryInfo crash when deleting docs

- Go
Published by zcdb almost 2 years ago

vearch - v3.4.3

Added

• add go sdk demo ### Removed

Changed

• check vector field and document_id for document/upsert ### Fixed
• fix multi filter search error, fix field range index boundary error
• fix multiple vector search
• fix override warning
• fix macOS compile error

- Go
Published by wxingda about 2 years ago

vearch - v3.4.2

Added

• add option to show hnsw search information
• add metric "vearchdiskstat"
• add interface for index/flush and index/forcemerge
• add move_member interface to move members
• support for modifying a single field
• add option to describe index information for interface of index/rebuild
• Add "space_name" to the engine to distinguish logs from different spaces
• add interface of index/rebuild
• add devtoolset-10 ### Removed
• remove return value of source and set return fields in unified way
• remove field source
• remove storage type "Mmap"
• remove zstd
• remove zfp ### Changed
• Change the storage type of the table to RocksDB
• print "meminfo" and "UnaryHandler" only when the LogInfoPrintSwitch switch is turned on
• check storetype and retrivaltype for index
• change indexed str limit to 1024 in router
• unified rocksdb version ### Fixed
• fix load of ivfpq index
• fix contex leak and reduce debug.Stack() calls
• fix bug when create vector table failed

- Go
Published by zcdb about 2 years ago

vearch - v3.4.1

Added

• add log for panic infomation
• add load_balance
• add option for check_restart
• add option to show detail info or not for cluster/health interface
• add resource check to prevent crash ### Removed

Changed

• extend STRMAXINDEX_LEN from 255 to 1024 ### Fixed
• modify fieldrangeindex delete fail log level
• Some index optimizations
• fix context leak
• limit fieldrangeindex_ size to prevent large memory alloc on loading

- Go
Published by wxingda over 2 years ago

vearch - v3.4.0

Added

• Add option of cluster_state in config
• Add node to faulty list on rpc error
• Add interface of document/query
• Add interface of document/upsert
• Add interface of document/search
• Add limit for document interface
• Add config option for replica_num ### Removed

Changed

• Redefine data interfaces
• Make rocksdb the default component
• Document/query support to get document on specify partition
• Unify the return format of document/delete interface
• Simplify the return data of document/upsert
• Unify the return format of document/[query|upsert] interface
• Update langchain plugins
• Update etcd version to v3.5.6
• Change etcd InitialClusterToken to cluster name
• Change itemtodocid_ storage to rocksdb ### Fixed
• Reformat codes and clear some warnnings
• Fix etcd import address

- Go
Published by wxingda over 2 years ago

vearch - v3.3.4

Added

• Add option for limited db num to one (https://github.com/vearch/vearch/commit/5756df13193d1661a316fb03dcaa46bdf4e0ee76) (https://github.com/vearch/vearch/commit/07f4e91dca4c802ee7ae6b6e8388bad59591e2c2)
• Add docker-image.yml (https://github.com/vearch/vearch/commit/5bb07da9aa2b5c7cc6ca3a6133845d5306c7a778)
• Add Authorization for interface of router (https://github.com/vearch/vearch/commit/7679e61cbd004d9edab10e94ad011f68914f577e) ### Removed

Changed

• Surport domain_name for etcd (https://github.com/vearch/vearch/commit/f0c1e2b995d54b192534f4a22bd65875a95e5fa5)
• Update process property array log (https://github.com/vearch/vearch/commit/b017ae3e5d0c7d49b4c8437d93b61bacdd605938)
• Add the _id return value of the get interface (https://github.com/vearch/vearch/commit/e1dcd76ce279e3bbc1cb0acec0d0ba409515d683) ### Fixed

- Go
Published by wxingda over 2 years ago

vearch - v3.3.4

Added

• Add option for limited db num to one (https://github.com/vearch/vearch/commit/5756df13193d1661a316fb03dcaa46bdf4e0ee76) (https://github.com/vearch/vearch/commit/07f4e91dca4c802ee7ae6b6e8388bad59591e2c2)
• Add docker-image.yml (https://github.com/vearch/vearch/commit/5bb07da9aa2b5c7cc6ca3a6133845d5306c7a778)
• Add Authorization for interface of router (https://github.com/vearch/vearch/commit/7679e61cbd004d9edab10e94ad011f68914f577e) ### Removed

Changed

• Surport domain_name for etcd](https://github.com/vearch/vearch/commit/f0c1e2b995d54b192534f4a22bd65875a95e5fa5)
• Update process property array log (https://github.com/vearch/vearch/commit/b017ae3e5d0c7d49b4c8437d93b61bacdd605938)
• Add the _id return value of the get interface (https://github.com/vearch/vearch/commit/e1dcd76ce279e3bbc1cb0acec0d0ba409515d683) ### Fixed

- Go
Published by wxingda over 2 years ago

vearch - v3.3.3

Added

• Add proxy master request in router (https://github.com/vearch/vearch/commit/07f4e91dca4c802ee7ae6b6e8388bad59591e2c2)
• Add GetMemoryInfo (https://github.com/vearch/vearch/commit/bd035ab26d6c138e8cf98c4fe36068effd2dd3ab)
• Add prefix for etcd key (https://github.com/vearch/vearch/commit/c60fdb3cb785953ed9d48126a57202bacf6a6a75)

Removed

• Remove BuildFieldIndex (https://github.com/vearch/vearch/commit/b5b00e005b1408292b81e879881219191ef5ef1d)

Changed

• Filter error items in handlerQueryDocByIdsFeature(https://github.com/vearch/vearch/commit/ce2b3d7941ba9a64c7eebcc5c617f5b5aa5345ff)
• Show handler cost time (https://github.com/vearch/vearch/commit/d611261f7ea178c389533eab81f4e43974241561)

Fixed

• Fix engine close MultiFieldsRangeIndex::FieldOperateWorker crash (https://github.com/vearch/vearch/commit/6dc2dd8ca38034cd9872015be13698e60c3fa4b1)
• Fix etcd version (https://github.com/vearch/vearch/commit/20502adc66bbf5b7c8f03e6988c654e0133090a9)

- Go
Published by zcdb over 2 years ago

vearch - v3.3.2

Added

• Add support for langchain (https://github.com/vearch/vearch/commit/a1b2d02e7613a3c3ebf61d434eea0aa91088d274)

Changed

• Set some default value for GammaVectorInfo in python sdk (https://github.com/vearch/vearch/commit/16792a008ffe643b3c819c5c8fe4715fe5d4e102)

Fixed

• Fix float32 filter error (https://github.com/vearch/vearch/commit/d8a6c261da7137a6243c6f5200c878a08894f67b)
• Fix schema index nil panic (https://github.com/vearch/vearch/commit/176fab90e8dcd8d1e9db2c4f7214f27989dddbc1)
• Fix python sdk compatibility (https://github.com/vearch/vearch/commit/b550395791b141fc984b178d940670429a38d21b)
• Fix HasBoost (https://github.com/vearch/vearch/commit/07b98c4f8bac5d7d13dff9f0af0a1d244bcf0ee8)

- Go
Published by zcdb over 2 years ago

vearch - v3.3.1

Added

• Filter maxStrLen from router (https://github.com/vearch/vearch/commit/e5bc5634e9c0d456f5807b88ca23e2f25d3fa8f3)
• Add interface to get doc through docid in the specified partition (https://github.com/vearch/vearch/commit/f1e3ce235bf5128dd9aff4491c179d18de27147e)

Removed

• Remove gammacb/lib (https://github.com/vearch/vearch/commit/6d13d227f0fb803ea4629ceb45985fc243efb972)

Changed

• Simplify python sdk function "create_table" (https://github.com/vearch/vearch/commit/07a83795c8fdabc049aaffe0220dbcac161fc62b)
• Load PS space meta info from master (https://github.com/vearch/vearch/commit/504e0177e205f75551714542fdafd299c18cb7a1)

Fixed

• Accelerate PS restart speed in place (https://github.com/vearch/vearch/commit/56fb09ebadd70c3fd65e45bfd713581a5450cdc1)
• Fix python sdk load err (https://github.com/vearch/vearch/commit/911c464f3ad91cd1186b81d35abcd36b98f58f3e)
• Check nprobe for ivfflat (https://github.com/vearch/vearch/commit/c47c6a5ca75c1b373567296fbe6ebbe75c43774a)
• Check update/delete id for hnsw (https://github.com/vearch/vearch/commit/553b06c84371a7f5987d1687e6dfb2de67d71180)
• Fix create table error which may cause ps crash (https://github.com/vearch/vearch/commit/952fb14ca375f65b000c51131f5a2647eeb6b618)

- Go
Published by wxingda over 2 years ago

vearch - v3.3.0

Added

• Add python demos
• Add more info in cluster/health and cluster/stats
• Add python sdk
• Add ps faulty list

Removed

• Deletebyquery no longer supports containing vectors

Changed

• Use rocksdb instead of mmap to persist memory raw vector

Fixed

• Fix raft error
• Fix deletebyquery
• Fix possible int overflow when vector dimension is large
• Fix null pointer when the search result is null
• Fix illegal json error when id type is long and the value is out of bound

- Go
Published by zcdb over 2 years ago

vearch -

1.Add option for build script. 2.HNSW now support brute force search. 3.Classification of gamma log.

- Go
Published by zcdb over 2 years ago

vearch - v3.2.9

1. Fixed build script.
2. Fixed continuous integration script.
3. Improved the speed of range index deletion.
4. Added test cases.
5. Integrated gamma into vearch.

- Go
Published by wxingda over 2 years ago

vearch - v3.2.8

Improve the performance of Python sdk; Add trace and timeout; Support router and ps resource isolation by resource name; Update vearch image search demo; Search replica using round robin instead of random strategy; Fix bugs of inserting and searching Compatible with faiss v1.7.1

- Go
Published by zcdb about 4 years ago

vearch - v3.2.7

Overview: Gamma supports three storage types: RocksDB, MemoryOnly, and Mmap. v3.2.7 mainly modifies and optimizes Mmap. The specific modifications are as follows:

1.Use a unified way to store vectors and scalars; 2.Use cache to improve query performance; 3.Reduce scalar memory usage; 4.Supports dynamic modification of the cache size, and flexible storage and retraction of memory; 5.Fix the occasional inaccurate query of scalar index 6.Support raft snapshot

- Go
Published by kuailelijuan almost 5 years ago

vearch -

1、Master API move to router，master can replace by etcd server。 need open config ： mastermergeinto_router。 2、Add search current limit function 3、Optimize loading logic 4、Replace faiss hnsw by hnswlib 5、Support the multiple retrieval models 6、Compatible with faiss 1.6.4 7、Update raft version 8、Some fix

- Go
Published by zcdb about 5 years ago

vearch - v3.2.0_1

1. fix compiling error without rocksdb in v3.2.0

- Go
Published by qiutianme over 5 years ago

vearch - v3.2.0 is released

1. The new interfaces of gamma engine retrieval model, ivfpq, ivfflat, ivfbinary and hnsw are all changed to the new interfaces.
2. Router, PS reconstruction: json changed to protobuf; Insert and search performance improved; Router added grpc interfaces.
3. Original vector compression, table string compression.
4. Master and etcd are separated, and the Master service supports automatic service discovery, query data node automatically shielded from accessing the failed node.

- Go
Published by wxingda over 5 years ago

vearch - v3.1.0 is released

1. Support real time HNSW index
2. Support binary index
3. Support IVFFLAT index
4. Fix bugs

- Go
Published by wxingda over 5 years ago