Indonesian Sentence Embeddings

Inspired by Thai Sentence Vector Benchmark, we decided to embark on the journey of training Indonesian sentence embedding models!

Evaluation

Semantic Textual Similarity

We followed approached done in the Thai Sentence Vector Benchmark project and translated the STS-B dev and test set to Indonesian via Google Translate API. This dataset will be used to evaluate our model's Spearman correlation score on the translated test set.

You can find the translated dataset on 🤗 HuggingFace Hub.

Further, we will similarly be evaluating our models on the SemRel2024 dataset which contains human-annotated, Indonesian semantic textual relatedness (STR) data. The dataset consists of two splits: dev and test. We will be evaluating our models' Spearman correlation score on both splits.

Retrieval

To evaluate our models' capability to perform retrieval tasks, we evaluate them on Indonesian subsets of MIRACL and TyDiQA datasets. In both datasets, the model's ability to retrieve relevant documents given a query is tested. We employ R@1 (top-1 accuracy), MRR@10, and nDCG@10 metrics to measure our model's performance.

Classification

For text classification, we will be doing emotion classification and sentiment analysis on the EmoT and SmSA subsets of IndoNLU, respectively. To do so, we will be doing the same approach as Thai Sentence Vector Benchmark and simply fit a Linear SVC on sentence representations of our texts with their corresponding labels. Thus, unlike conventional fine-tuning method where the backbone model is also updated, the Sentence Transformer stays frozen in our case; with only the classification head being trained.

Further, we will evaluate our models using the official MTEB code that contains two Indonesian classification subtasks: MassiveIntentClassification (id) and MassiveScenarioClassification (id).

Pair Classification

We followed MTEB's PairClassification evaluation procedure for pair classification. Specifically for zero-shot natural language inference tasks, all neutral pairs are dropped, while contradictions and entailments are re-mapped as 0s and 1s. The maximum average precision (AP) score is found by finding the best threshold value.

We leverage the IndoNLI dataset's two test subsets: test_lay and test_expert.

Methods

(Unsupervised) SimCSE

We followed SimCSE: Simple Contrastive Learning of Sentence Embeddings and trained a sentence embedding model in an unsupervised fashion. Unsupervised SimCSE allows us to leverage an unsupervised corpus -- which are plenty -- and with different dropout masks in the encoder, contrastively learn sentence representations. This is parallel with the situation that there is a lack of supervised Indonesian sentence similarity datasets, hence SimCSE is a natural first move into this field. We used the Sentence Transformer implementation of SimCSE.

ConGen

Like SimCSE, ConGen: Unsupervised Control and Generalization Distillation For Sentence Representation is another unsupervised technique to train a sentence embedding model. Since it is in-part a distillation method, ConGen relies on a teacher model which will then be distilled to a student model. The original paper proposes back-translation as the best data augmentation technique. However, due to the lack of resources, we implemented word deletion, which was found to be on-par with back-translation despite being trivial. We used the official ConGen implementation which was written on top of the Sentence Transformers library.

SCT

SCT: An Efficient Self-Supervised Cross-View Training For Sentence Embedding is another unsupervised technique to train a sentence embedding model. It is very similar to ConGen in its knowledge distillation methodology, but also supports self-supervised training procedure without a teacher model. The original paper proposes back-translation as its data augmentation technique, but we implemented single-word deletion and found it to perform better than our backtranslated corpus. We used the official SCT implementation which was written on top of the Sentence Transformers library.

Pretrained Models

| Model | #params | Base/Student Model | Teacher Model | Train Dataset | Supervised | | --------------------------------------------------------------------------------------------------------------------------- | :-----: | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | :--------: | | SimCSE-IndoBERT Base | 125M | IndoBERT Base | N/A | Wikipedia | | | ConGen-IndoBERT Lite Base | 12M | IndoBERT Lite Base | paraphrase-multilingual-mpnet-base-v2 | Wikipedia | | | ConGen-IndoBERT Base | 125M | IndoBERT Base | paraphrase-multilingual-mpnet-base-v2 | Wikipedia | | | ConGen-SimCSE-IndoBERT Base | 125M | SimCSE-IndoBERT Base | paraphrase-multilingual-mpnet-base-v2 | Wikipedia | | | ConGen-Indo-e5 Small | 118M | multilingual-e5-small | paraphrase-multilingual-mpnet-base-v2 | Wikipedia | | | SCT-IndoBERT Base | 125M | IndoBERT Base | paraphrase-multilingual-mpnet-base-v2 | Wikipedia | | | all-IndoBERT Base | 125M | IndoBERT Base | N/A | See: README | ✅ | | all-IndoBERT Base-v2 | 125M | IndoBERT Base | N/A | See: README | ✅ | | all-IndoBERT Base-v4 | 125M | IndoBERT Base | N/A | See: README | ✅ | | all-NusaBERT Base-v4 | 111M | NusaBERT Base | N/A | See: README | ✅ | | all-NusaBERT Large-v4 | 337M | NusaBERT Large | N/A | See: README | ✅ | | all-Indo-e5 Small-v2 | 118M | multilingual-e5-small | N/A | See: README | ✅ | | all-Indo-e5 Small-v3 | 118M | multilingual-e5-small | N/A | See: README | ✅ | | all-Indo-e5 Small-v4 | 118M | multilingual-e5-small | N/A | See: README | ✅ | | distiluse-base-multilingual-cased-v2 | 134M | DistilBERT Base Multilingual | mUSE | See: SBERT | ✅ | | paraphrase-multilingual-mpnet-base-v2 | 125M | XLM-RoBERTa Base | paraphrase-mpnet-base-v2 | See: SBERT | ✅ | | multilingual-e5-small | 118M | Multilingual-MiniLM-L12-H384 | See: arXiv | See: 🤗 | ✅ | | multilingual-e5-base | 278M | XLM-RoBERTa Base | See: arXiv | See: 🤗 | ✅ | | multilingual-e5-large | 560M | XLM-RoBERTa Large | See: arXiv | See: 🤗 | ✅ |

Results

Semantic Textual Similarity

Machine Translated Indonesian STS-B

| Model | Spearman's Correlation (%) ↑ | | --------------------------------------------------------------------------------------------------------------------------- | :--------------------------: | | SimCSE-IndoBERT Base | 70.13 | | ConGen-IndoBERT Lite Base | 79.97 | | ConGen-IndoBERT Base | 80.47 | | ConGen-SimCSE-IndoBERT Base | 81.16 | | ConGen-Indo-e5 Small | 80.94 | | SCT-IndoBERT Base | 74.56 | | all-IndoBERT Base | 73.84 | | all-IndoBERT Base-v2 | 76.03 | | all-IndoBERT Base-v4 | 75.99 | | all-NusaBERT Base-v4 | 77.65 | | all-NusaBERT Large-v4 | 79.23 | | all-Indo-e5 Small-v2 | 79.57 | | all-Indo-e5 Small-v3 | 79.95 | | all-Indo-e5 Small-v4 | 79.85 | | distiluse-base-multilingual-cased-v2 | 75.08 | | paraphrase-multilingual-mpnet-base-v2 | 83.83 | | multilingual-e5-small | 78.89 | | multilingual-e5-base | 79.72 | | multilingual-e5-large | 79.44 |

SemRel2024: Semantic Textual Relatedness (STR)

| Model | dev Spearman's Correlation (%) ↑ | test Spearman's Correlation (%) ↑ | | --------------------------------------------------------------------------------------------------------------------------- | :--------------------------------: | :---------------------------------: | | SimCSE-IndoBERT Base | 30.64 | 36.77 | | ConGen-IndoBERT Lite Base | 35.95 | 41.73 | | ConGen-IndoBERT Base | 35.05 | 39.14 | | ConGen-SimCSE-IndoBERT Base | 33.71 | 37.73 | | ConGen-Indo-e5 Small | 36.35 | 42.47 | | SCT-IndoBERT Base | 41.50 | 43.25 | | all-IndoBERT Base | 42.87 | 38.78 | | all-IndoBERT Base-v2 | 41.68 | 40.42 | | all-IndoBERT Base-v4 | 41.38 | 38.05 | | all-NusaBERT Base-v4 | 42.11 | 41.55 | | all-NusaBERT Large-v4 | 40.21 | 42.25 | | all-Indo-e5 Small-v2 | 39.79 | 43.85 | | all-Indo-e5 Small-v3 | 40.25 | 42.60 | | all-Indo-e5 Small-v4 | 40.20 | 42.90 | | distiluse-base-multilingual-cased-v2 | 37.22 | 49.35 | | paraphrase-multilingual-mpnet-base-v2 | 34.56 | 37.51 | | multilingual-e5-small | 41.92 | 49.60 | | multilingual-e5-base | 41.29 | 45.04 | | multilingual-e5-large | 39.20 | 45.04 |

Retrieval

MIRACL

| Model | R@1 (%) ↑ | MRR@10 (%) ↑ | nDCG@10 (%) ↑ | | --------------------------------------------------------------------------------------------------------------------------- | :-------: | :----------: | :-----------: | | SimCSE-IndoBERT Base | 36.04 | 48.25 | 39.70 | | ConGen-IndoBERT Lite Base | 46.04 | 59.06 | 51.01 | | ConGen-IndoBERT Base | 45.93 | 58.58 | 49.95 | | ConGen-SimCSE-IndoBERT Base | 45.83 | 58.27 | 49.91 | | ConGen-Indo-e5 Small | 55.00 | 66.74 | 58.95 | | SCT-IndoBERT Base | 40.41 | 47.29 | 40.68 | | all-IndoBERT Base | 65.52 | 75.92 | 70.13 | | all-IndoBERT Base-v2 | 67.18 | 76.59 | 70.16 | | all-IndoBERT Base-v4 | 67.91 | 77.37 | 70.97 | | all-NusaBERT Base-v4 | 67.08 | 77.47 | 71.24 | | all-NusaBERT Large-v4 | 68.43 | 78.29 | 71.99 | | all-Indo-e5 Small-v2 | 68.33 | 78.33 | 73.04 | | all-Indo-e5 Small-v3 | 68.12 | 78.22 | 73.09 | | all-Indo-e5 Small-v4 | 68.33 | 78.41 | 73.23 | | distiluse-base-multilingual-cased-v2 | 41.35 | 54.93 | 48.79 | | paraphrase-multilingual-mpnet-base-v2 | 52.81 | 65.07 | 57.97 | | multilingual-e5-small | 70.20 | 79.61 | 74.80 | | multilingual-e5-base | 70.00 | 79.50 | 75.16 | | multilingual-e5-large | 70.83 | 80.58 | 76.16 |

TyDiQA

| Model | R@1 (%) ↑ | MRR@10 (%) ↑ | nDCG@10 (%) ↑ | | --------------------------------------------------------------------------------------------------------------------------- | :-------: | :----------: | :-----------: | | SimCSE-IndoBERT Base | 61.94 | 69.89 | 73.52 | | ConGen-IndoBERT Lite Base | 75.22 | 81.55 | 84.13 | | ConGen-IndoBERT Base | 73.09 | 80.32 | 83.29 | | ConGen-SimCSE-IndoBERT Base | 72.38 | 79.37 | 82.51 | | ConGen-Indo-e5 Small | 84.60 | 89.30 | 91.27 | | SCT-IndoBERT Base | 76.81 | 83.16 | 85.87 | | all-IndoBERT Base | 88.14 | 91.47 | 92.91 | | all-IndoBERT Base-v2 | 87.61 | 90.91 | 92.31 | | all-IndoBERT Base-v4 | 89.02 | 92.59 | 93.91 | | all-NusaBERT Base-v4 | 92.74 | 94.95 | 95.73 | | all-NusaBERT Large-v4 | 93.62 | 95.77 | 96.56 | | all-Indo-e5 Small-v2 | 93.27 | 95.63 | 96.46 | | all-Indo-e5 Small-v3 | 93.27 | 95.72 | 96.58 | | all-Indo-e5 Small-v4 | 93.45 | 95.66 | 96.43 | | distiluse-base-multilingual-cased-v2 | 70.44 | 77.94 | 81.56 | | paraphrase-multilingual-mpnet-base-v2 | 81.41 | 87.05 | 89.44 | | multilingual-e5-small | 91.50 | 94.34 | 95.39 | | multilingual-e5-base | 93.45 | 95.88 | 96.69 | | multilingual-e5-large | 94.69 | 96.71 | 97.44 |

Classification

MTEB - Massive Intent Classification (id)

| Model | Accuracy (%) ↑ | F1 Macro (%) ↑ | | --------------------------------------------------------------------------------------------------------------------------- | :------------: | :------------: | | SimCSE-IndoBERT Base | 59.71 | 57.70 | | ConGen-IndoBERT Lite Base | 62.41 | 60.94 | | ConGen-IndoBERT Base | 61.14 | 60.02 | | ConGen-SimCSE-IndoBERT Base | 60.93 | 59.50 | | ConGen-Indo-e5 Small | 62.92 | 60.18 | | SCT-IndoBERT Base | 55.66 | 54.48 | | all-IndoBERT Base | 58.40 | 57.21 | | all-IndoBERT Base-v2 | 58.31 | 57.11 | | all-IndoBERT Base-v4 | 57.80 | 56.71 | | all-NusaBERT Base-v4 | 62.10 | 60.38 | | all-NusaBERT Large-v4 | 61.41 | 59.93 | | all-Indo-e5 Small-v2 | 61.51 | 59.24 | | all-Indo-e5 Small-v3 | 61.63 | 59.29 | | all-Indo-e5 Small-v4 | 61.38 | 59.07 | | distiluse-base-multilingual-cased-v2 | 55.99 | 52.44 | | paraphrase-multilingual-mpnet-base-v2 | 65.43 | 63.55 | | multilingual-e5-small | 64.16 | 61.33 | | multilingual-e5-base | 66.63 | 63.88 | | multilingual-e5-large | 70.04 | 67.66 |

MTEB - Massive Scenario Classification (id)

| Model | Accuracy (%) ↑ | F1 Macro (%) ↑ | | --------------------------------------------------------------------------------------------------------------------------- | :------------: | :------------: | | SimCSE-IndoBERT Base | 66.14 | 65.56 | | ConGen-IndoBERT Lite Base | 67.25 | 66.53 | | ConGen-IndoBERT Base | 67.72 | 67.32 | | ConGen-SimCSE-IndoBERT Base | 67.12 | 66.64 | | ConGen-Indo-e5 Small | 66.92 | 66.29 | | SCT-IndoBERT Base | 61.89 | 60.97 | | all-IndoBERT Base | 66.37 | 66.31 | | all-IndoBERT Base-v2 | 66.02 | 65.97 | | all-IndoBERT Base-v4 | 66.33 | 66.14 | | all-NusaBERT Base-v4 | 70.17 | 70.18 | | all-NusaBERT Large-v4 | 70.10 | 70.38 | | all-Indo-e5 Small-v2 | 67.02 | 66.86 | | all-Indo-e5 Small-v3 | 67.27 | 67.13 | | all-Indo-e5 Small-v4 | 67.33 | 67.24 | | distiluse-base-multilingual-cased-v2 | 65.25 | 63.45 | | paraphrase-multilingual-mpnet-base-v2 | 70.72 | 70.58 | | multilingual-e5-small | 67.92 | 67.23 | | multilingual-e5-base | 70.70 | 70.26 | | multilingual-e5-large | 74.11 | 73.82 |

IndoNLU - Emotion Classification (EmoT)

| Model | Accuracy (%) ↑ | F1 Macro (%) ↑ | | --------------------------------------------------------------------------------------------------------------------------- | :------------: | :------------: | | SimCSE-IndoBERT Base | 55.45 | 55.78 | | ConGen-IndoBERT Lite Base | 58.18 | 58.84 | | ConGen-IndoBERT Base | 57.04 | 57.06 | | ConGen-SimCSE-IndoBERT Base | 59.54 | 60.37 | | ConGen-Indo-e5 Small | 60.00 | 60.52 | | SCT-IndoBERT Base | 61.13 | 61.70 | | all-IndoBERT Base | 57.27 | 57.47 | | all-IndoBERT Base-v2 | 58.86 | 59.31 | | all-IndoBERT Base-v4 | 61.36 | 61.81 | | all-NusaBERT Base-v4 | 53.18 | 53.01 | | all-NusaBERT Large-v4 | 63.18 | 63.17 | | all-Indo-e5 Small-v2 | 58.18 | 57.99 | | all-Indo-e5 Small-v3 | 56.81 | 56.46 | | all-Indo-e5 Small-v4 | 56.94 | 57.04 | | distiluse-base-multilingual-cased-v2 | 63.63 | 64.13 | | paraphrase-multilingual-mpnet-base-v2 | 63.18 | 63.78 | | multilingual-e5-small | 64.54 | 65.04 | | multilingual-e5-base | 68.63 | 69.07 | | multilingual-e5-large | 74.77 | 74.66 |

IndoNLU - Sentiment Analysis (SmSA)

| Model | Accuracy (%) ↑ | F1 Macro (%) ↑ | | --------------------------------------------------------------------------------------------------------------------------- | :------------: | :------------: | | SimCSE-IndoBERT Base | 85.6 | 81.50 | | ConGen-IndoBERT Lite Base | 81.2 | 75.59 | | ConGen-IndoBERT Base | 85.4 | 82.12 | | ConGen-SimCSE-IndoBERT Base | 83.0 | 78.74 | | ConGen-Indo-e5 Small | 84.2 | 80.21 | | SCT-IndoBERT Base | 82.0 | 76.92 | | all-IndoBERT Base | 84.4 | 79.79 | | all-IndoBERT Base-v2 | 83.4 | 79.04 | | all-IndoBERT Base-v4 | 82.4 | 77.82 | | all-NusaBERT Base-v4 | 84.2 | 78.68 | | all-NusaBERT Large-v4 | 84.8 | 81.01 | | all-Indo-e5 Small-v2 | 82.0 | 78.15 | | all-Indo-e5 Small-v3 | 82.6 | 78.98 | | all-Indo-e5 Small-v4 | 82.6 | 79.14 | | distiluse-base-multilingual-cased-v2 | 78.8 | 73.64 | | paraphrase-multilingual-mpnet-base-v2 | 89.6 | 86.56 | | multilingual-e5-small | 83.6 | 79.51 | | multilingual-e5-base | 89.4 | 86.22 | | multilingual-e5-large | 90.0 | 86.50 |

Pair Classification

IndoNLI

| Model | test_lay AP (%) ↑ | test_expert AP (%) ↑ | | --------------------------------------------------------------------------------------------------------------------------- | :-----------------: | :--------------------: | | SimCSE-IndoBERT Base | 56.06 | 50.72 | | ConGen-IndoBERT Lite Base | 69.44 | 53.74 | | ConGen-IndoBERT Base | 71.14 | 56.35 | | ConGen-SimCSE-IndoBERT Base | 70.80 | 56.59 | | ConGen-Indo-e5 Small | 70.51 | 55.67 | | SCT-IndoBERT Base | 59.82 | 53.41 | | all-IndoBERT Base | 72.01 | 56.79 | | all-IndoBERT Base-v2 | 71.36 | 56.83 | | all-IndoBERT Base-v4 | 70.99 | 58.99 | | all-NusaBERT Base-v4 | 73.07 | 59.86 | | all-NusaBERT Large-v4 | 73.26 | 61.14 | | all-Indo-e5 Small-v2 | 76.29 | 57.05 | | all-Indo-e5 Small-v3 | 75.21 | 56.62 | | all-Indo-e5 Small-v4 | 75.05 | 57.42 | | distiluse-base-multilingual-cased-v2 | 58.48 | 50.50 | | paraphrase-multilingual-mpnet-base-v2 | 74.87 | 57.96 | | multilingual-e5-small | 63.97 | 51.85 | | multilingual-e5-base | 60.25 | 50.91 | | multilingual-e5-large | 61.39 | 51.62 |

Credits

Indonesian Sentence Embeddings is developed with love by: