https://github.com/beomi/kobertscore

BERTScore for Korean

https://github.com/beomi/kobertscore

Repository

BERTScore for Korean

https://github.com/Beomi/KoBERTScore/blob/master/

# Ko-BERTScore

BERTScore using pretrained Korean BERT. This package provides re-implemented version of BERTScore.

## Install

```
git clone https://github.com/lovit/KoBERTScore
cd ko-BERTScore
python setup.py install
```

## Usage

### Evaluate sentence pairs

Using BERTScore class instance

```python
from KoBERTScore import BERTScore

model_name = "beomi/kcbert-base"
bertscore = BERTScore(model_name, best_layer=4)

references = [
    '      ',
    '   ',
    '   ',
    '    '
]
candidates = [
    '      ',
    '   ',
    '      ',
    '? ?'
]

bertscore(references, candidates, batch_size=128)
# [0.5643115, 0.4720116, 0.2556618, 0.2268927]
```

Using manually loaded BERT model

```python
from transformers import BertModel, BertTokenizer

model_name = "bert-base-uncased"
tokenizer = BertTokenizer.from_pretrained(model_name)
encoder = BertModel.from_pretrained(model_name)
pretrained_idf_embedding # torch.nn.Embedding

references = ['hello world', 'my name is lovit', 'oh hi', 'where I am', 'where we are going']
candidates = ['Hellow words', 'I am lovit', 'oh hello', 'where am I', 'where we go']
bert_score(bert_tokenizer, bert_model, references, candidates)
bert_score(bert_tokenizer, bert_model, references, candidates, idf=pretrained_idf_embedding)
```

### Draw pairwise cosine similarity of bert embedding

Using BERTScore class instance

```python
from KoBERTScore import BERTScore
from bokeh.plotting import show

reference = '      '
candidate = '      '
bertscore = BERTScore()  # default model is 'beomi/kcbert-base'
p = bertscore.plot_bertscore_detail(reference, candidate)
show(p)
```

Loading BERT manually

```python
from transformers import BertModel, BertTokenizer
from bokeh.plotting import show, output_notebook
from KoBERTScore import plot_bertscore_detail

model_name = 'beomi/kcbert-base'
tokenizer = BertTokenizer.from_pretrained(model_name)
encoder = BertModel.from_pretrained(model_name)

reference = '      '
candidate = '      '
p = plot_bertscore_detail(reference, candidate, tokenizer, encoder)

# output_notebook()  # If env is IPython notebook
show(p)
```

![](resources/bertscore_pairwise_cosine.png)

### Finding best layer
```
kobertscore best_layer \
  --corpus korsts \
  --model_name_or_path beomi/kcbert-base \
  --draw_plot \
  --output_dir .
```

### Finding rescale base
```
kobertscore rescale_base \
  --model_name_or_path beomi/kcbert-base \
  --references path/to/references.txt \
  --output_path MODEL_NAME_base
```

### Compute average L2 norm of every BERT layer output
```
kobertscore l2norm \
  --model_name_or_path beomi/kcbert-base \
  --references path/to/references \
  --output_path kcbert-l2norm \
  --draw_plot
```

## Performance and best-layer index of Korean BERT models

Tested correlation between BERTScore and [KorSTS](https://github.com/ko-nlp/Korpora#korsts) score

| model | layer index | correlation (F)|
| --- | --- | --- |
| 'beomi/kcbert-base' | 4 | 0.622 |
| 'monologg/distilkobert' | 12 | 0.334 |
| 'monologg/kobert' | 2 | 0.190 |
| 'monologg/koelectra-base-v2-discriminator' | 12 | 0.098 |

![](resources/kcbert_korsts.png)

## Reference
- Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). [Bertscore: Evaluating text generation with bert.](https://arxiv.org/abs/1904.09675) arXiv preprint arXiv:1904.09675.

Owner

GitHub Events