Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of text2vec

text2vec - 1.2.9

1.2.9版本

  1. 支持多卡推理（多进程实现多GPU、多CPU推理），text2vec支持多卡推理(计算文本向量): examples/computingembeddingsmultigpudemo.py

  2. 新增命令行工具（CLI），可以无需代码开发批量获取文本向量： zsh pip install text2vec>=1.2.9 text2vec --input_file input.txt --output_file out.csv --batch_size 128 --multi_gpu True

Full Changelog: https://github.com/shibing624/text2vec/compare/1.2.8...1.2.9

- Python
Published by shibing624 over 2 years ago

text2vec - 1.2.8

1.2.8版本

  1. 支持多卡推理（多进程实现多GPU和多CPU推理），text2vec支持多卡推理(计算文本向量): examples/computingembeddingsmultigpudemo.py

  2. 新增命令行工具（CLI），可以无需代码开发批量获取文本向量： zsh pip install text2vec -U text2vec --input_file input.txt --output_file out.csv --batch_size 16

Full Changelog: https://github.com/shibing624/text2vec/compare/1.2.4...1.2.8

- Python
Published by shibing624 over 2 years ago

text2vec - 1.2.4

v1.2.4版本

  1. 实现了BGE微调训练方法 ，支持自定义样本集训练 https://github.com/shibing624/text2vec/blob/master/examples/trainingbgemodelmydata.py ；支持构建训练样本集 https://github.com/shibing624/text2vec/blob/master/examples/data/buildzhbgedataset.py ；支持使用C-MTEB评估 https://github.com/shibing624/text2vec/blob/master/tests/eval_C-MTEB.py
  2. 发布了中文匹配模型shibing624/text2vec-bge-large-chinese，用CoSENT方法训练，基于BAAI/bge-large-zh-noinstruct用人工挑选后的中文STS数据集shibing624/nli-zh-all/text2vec-base-chinese-paraphrase-dataset训练得到，并在中文测试集评估相对于原模型效果有提升，相较于原模型在短文本区分度上提升明显。

Full Changelog: https://github.com/shibing624/text2vec/compare/1.2.3...1.2.4

- Python
Published by shibing624 over 2 years ago

text2vec - v1.2.2

### v1.2.2版本

英文匹配数据集的评测结果：

| Arch | BaseModel | Model | English-STS-B | |:-------|:------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------|:-------------:| | GloVe | glove | Avgwordembeddingsglove6B300d | 61.77 | | BERT | bert-base-uncased | BERT-base-cls | 20.29 | | BERT | bert-base-uncased | BERT-base-firstlastavg | 59.04 | | BERT | bert-base-uncased | BERT-base-firstlastavg-whiten(NLI) | 63.65 | | SBERT | sentence-transformers/bert-base-nli-mean-tokens | SBERT-base-nli-cls | 73.65 | | SBERT | sentence-transformers/bert-base-nli-mean-tokens | SBERT-base-nli-firstlastavg | 77.96 | | CoSENT | bert-base-uncased | CoSENT-base-firstlastavg | 69.93 | | CoSENT | sentence-transformers/bert-base-nli-mean-tokens | CoSENT-base-nli-firstlast_avg | 79.68 | | CoSENT | sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 | shibing624/text2vec-base-multilingual | 80.12 |

  • 本项目release模型的中文匹配评测结果：

| Arch | BaseModel | Model | ATEC | BQ | LCQMC | PAWSX | STS-B | SOHU-dd | SOHU-dc | Avg | QPS | |:-----------|:-------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------|:-----:|:-----:|:-----:|:-----:|:-----:|:-------:|:-------:|:---------:|:-----:| | Word2Vec | word2vec | w2v-light-tencent-chinese | 20.00 | 31.49 | 59.46 | 2.57 | 55.78 | 55.04 | 20.70 | 35.03 | 23769 | | SBERT | xlm-roberta-base | sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 | 18.42 | 38.52 | 63.96 | 10.14 | 78.90 | 63.01 | 52.28 | 46.46 | 3138 | | Instructor | hfl/chinese-roberta-wwm-ext | moka-ai/m3e-base | 41.27 | 63.81 | 74.87 | 12.20 | 76.96 | 75.83 | 60.55 | 57.93 | 2980 | | CoSENT | hfl/chinese-macbert-base | shibing624/text2vec-base-chinese | 31.93 | 42.67 | 70.16 | 17.21 | 79.30 | 70.27 | 50.42 | 51.61 | 3008 | | CoSENT | hfl/chinese-lert-large | GanymedeNil/text2vec-large-chinese | 32.61 | 44.59 | 69.30 | 14.51 | 79.44 | 73.01 | 59.04 | 53.12 | 2092 | | CoSENT | nghuyong/ernie-3.0-base-zh | shibing624/text2vec-base-chinese-sentence | 43.37 | 61.43 | 73.48 | 38.90 | 78.25 | 70.60 | 53.08 | 59.87 | 3089 | | CoSENT | nghuyong/ernie-3.0-base-zh | shibing624/text2vec-base-chinese-paraphrase | 44.89 | 63.58 | 74.24 | 40.90 | 78.93 | 76.70 | 63.30 | 63.08 | 3066 | | CoSENT | sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 | shibing624/text2vec-base-multilingual | 32.39 | 50.33 | 65.64 | 32.56 | 74.45 | 68.88 | 51.17 | 53.67 | 4004 |

说明： - 结果评测指标：spearman系数 - shibing624/text2vec-base-chinese模型，是用CoSENT方法训练，基于hfl/chinese-macbert-base在中文STS-B数据训练得到，并在中文STS-B测试集评估达到较好效果，运行examples/trainingsuptextmatchingmodel.py代码可训练模型，模型文件已经上传HF model hub，中文通用语义匹配任务推荐使用 - shibing624/text2vec-base-chinese-sentence模型，是用CoSENT方法训练，基于nghuyong/ernie-3.0-base-zh用人工挑选后的中文STS数据集shibing624/nli-zh-all/text2vec-base-chinese-sentence-dataset训练得到，并在中文各NLI测试集评估达到较好效果，运行examples/trainingsuptextmatchingmodeljsonldata.py代码可训练模型，模型文件已经上传HF model hub，中文s2s(句子vs句子)语义匹配任务推荐使用 - shibing624/text2vec-base-chinese-paraphrase模型，是用CoSENT方法训练，基于nghuyong/ernie-3.0-base-zh用人工挑选后的中文STS数据集shibing624/nli-zh-all/text2vec-base-chinese-paraphrase-dataset，数据集相对于shibing624/nli-zh-all/text2vec-base-chinese-sentence-dataset加入了s2p(sentence to paraphrase)数据，强化了其长文本的表征能力，并在中文各NLI测试集评估达到SOTA，运行examples/trainingsuptextmatchingmodeljsonldata.py代码可训练模型，模型文件已经上传HF model hub，中文s2p(句子vs段落)语义匹配任务推荐使用 - shibing624/text2vec-base-multilingual模型，是用CoSENT方法训练，基于sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2用人工挑选后的多语言STS数据集shibing624/nli-zh-all/text2vec-base-multilingual-dataset训练得到，并在中英文测试集评估相对于原模型效果有提升，运行examples/trainingsuptextmatchingmodeljsonldata.py代码可训练模型，模型文件已经上传HF model hub，多语言语义匹配任务推荐使用

Full Changelog: https://github.com/shibing624/text2vec/compare/1.2.1...1.2.2

- Python
Published by shibing624 over 2 years ago

text2vec - v1.2.1

v1.2.1

Release Models

  • 本项目release模型的中文匹配评测结果：

| Arch | BaseModel | Model | ATEC | BQ | LCQMC | PAWSX | STS-B | SOHU-dd | SOHU-dc | Avg | QPS | |:-----------|:----------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------|:-----:|:-----:|:-----:|:-----:|:-----:|:-------:|:-------:|:---------:|:-----:| | Word2Vec | word2vec | w2v-light-tencent-chinese | 20.00 | 31.49 | 59.46 | 2.57 | 55.78 | 55.04 | 20.70 | 35.03 | 23769 | | SBERT | xlm-roberta-base | sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 | 18.42 | 38.52 | 63.96 | 10.14 | 78.90 | 63.01 | 52.28 | 46.46 | 3138 | | Instructor | hfl/chinese-roberta-wwm-ext | moka-ai/m3e-base | 41.27 | 63.81 | 74.87 | 12.20 | 76.96 | 75.83 | 60.55 | 57.93 | 2980 | | CoSENT | hfl/chinese-macbert-base | shibing624/text2vec-base-chinese | 31.93 | 42.67 | 70.16 | 17.21 | 79.30 | 70.27 | 50.42 | 51.61 | 3008 | | CoSENT | hfl/chinese-lert-large | GanymedeNil/text2vec-large-chinese | 32.61 | 44.59 | 69.30 | 14.51 | 79.44 | 73.01 | 59.04 | 53.12 | 2092 | | CoSENT | nghuyong/ernie-3.0-base-zh | shibing624/text2vec-base-chinese-sentence | 43.37 | 61.43 | 73.48 | 38.90 | 78.25 | 70.60 | 53.08 | 59.87 | 3089 | | CoSENT | nghuyong/ernie-3.0-base-zh | shibing624/text2vec-base-chinese-paraphrase | 44.89 | 63.58 | 74.24 | 40.90 | 78.93 | 76.70 | 63.30 | 63.08 | 3066 |

Full Changelog: https://github.com/shibing624/text2vec/compare/1.2.0...1.2.1

- Python
Published by shibing624 over 2 years ago

text2vec - v1.2.0

v1.2.0版本

  • 发布了中文匹配模型shibing624/text2vec-base-chinese-nli，基于ERNIE-3.0-base模型，使用了中文NLI数据集shibing624/nli_zh全部语料训练的CoSENT文本匹配模型，在各评估集表现提升明显。

  • 发布了2个中文NLI数据集：shibing624/snli-zh 和 shibing624/nli-zh-all

  • 本项目release模型的中文匹配评测结果：

| Arch | BaseModel | Model | ATEC | BQ | LCQMC | PAWSX | STS-B | Avg | QPS | | :-- |:-----------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|:-----:| | Word2Vec | word2vec | w2v-light-tencent-chinese | 20.00 | 31.49 | 59.46 | 2.57 | 55.78 | 33.86 | 23769 | | SBERT | xlm-roberta-base | sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 | 18.42 | 38.52 | 63.96 | 10.14 | 78.90 | 41.99 | 3138 | | CoSENT | hfl/chinese-macbert-base | shibing624/text2vec-base-chinese | 31.93 | 42.67 | 70.16 | 17.21 | 79.30 | 48.25 | 3008 | | CoSENT | hfl/chinese-lert-large | GanymedeNil/text2vec-large-chinese | 32.61 | 44.59 | 69.30 | 14.51 | 79.44 | 48.08 | 2092 | | CoSENT | nghuyong/ernie-3.0-base-zh | shibing624/text2vec-base-chinese-nli | 51.26 | 68.72 | 79.13 | 34.28 | 80.70 | 62.81 | 3066 |

  • 本项目release的数据集：

| Dataset | Introduce | Download Link | |:----------------------|:--------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | shibing624/nli-zh-all | 中文语义匹配数据合集，整合了文本推理，相似，摘要，问答，指令微调等任务的820万高质量数据，并转化为匹配格式数据集 | https://huggingface.co/datasets/shibing624/nli-zh-all | | shibing624/snli-zh | 中文SNLI和MultiNLI数据集，翻译自英文SNLI和MultiNLI | https://huggingface.co/datasets/shibing624/snli-zh | | shibing624/nli_zh | 中文语义匹配数据集，整合了中文ATEC、BQ、LCQMC、PAWSX、STS-B共5个任务的数据集 | https://huggingface.co/datasets/shibing624/nli_zh
or
百度网盘(提取码:qkt6)
or
github
|

  • 基于更大数据集shibing624/nli-zh-all的CoSENT匹配模型在训练中。

Full Changelog: https://github.com/shibing624/text2vec/compare/1.1.8...1.2.0

- Python
Published by shibing624 over 2 years ago

text2vec - 1.1.4

v1.1.4版本

发布了中文匹配模型shibing624/text2vec-base-chinese，基于中文STS训练集训练的CoSENT匹配模型。

  • 本项目release模型的中文匹配评测结果：

| Arch | BaseModel | Model | ATEC | BQ | LCQMC | PAWSX | STS-B | Avg | QPS | | :-- |:-----------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------|:-----:|:-----:|:-----:|:-----:|:-----:|:---------:|:-----:| | Word2Vec | word2vec | w2v-light-tencent-chinese | 20.00 | 31.49 | 59.46 | 2.57 | 55.78 | 33.86 | 23769 | | SBERT | xlm-roberta-base | sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 | 18.42 | 38.52 | 63.96 | 10.14 | 78.90 | 41.99 | 3138 | | CoSENT | hfl/chinese-macbert-base | shibing624/text2vec-base-chinese | 31.93 | 42.67 | 70.16 | 17.21 | 79.30 | 48.25 | 3008 |

Full Changelog: https://github.com/shibing624/text2vec/compare/1.1.3...1.1.4

add word2vec tencent light embeddings file: lightTencentAILab_ChineseEmbedding.bin

- Python
Published by shibing624 almost 4 years ago

text2vec - 1.1.3

Full Changelog: https://github.com/shibing624/text2vec/compare/1.1.2...1.1.3

- Python
Published by shibing624 almost 4 years ago

text2vec - 1.1.2

add dataset of nli_zh

- Python
Published by shibing624 almost 4 years ago

text2vec - 1.1.0

重写了CoSENT, SentenceBERT模型的训练和预测代码： 1. 句子匹配模型训练逻辑继承基类SentenceModel， 2. 新增trainmodel, evalmodel, 代码结构更清晰， 3. 预测均使用基类的encode实现。

- Python
Published by shibing624 almost 4 years ago

text2vec - 0.1.3

0.1.3, new CoSENT model.

- Python
Published by shibing624 about 4 years ago

text2vec -

- Python
Published by shibing624 almost 5 years ago