https://github.com/beomi/kor2vec

OOV없이 빠르고 정확한 한국어 Embedding 라이브러리

https://github.com/beomi/kor2vec

Repository

OOV없이 빠르고 정확한 한국어 Embedding 라이브러리

https://github.com/Beomi/kor2vec/blob/master/

# kor2vec

OOV    Embedding

## Installation
```shell
pip install git+https://github.com/naver/kor2vec.git
```
> Requirements : `tqdm`, `numpy` and support `torch >= 0.4.0`

## Introduction

    .  +(), +    
     .      
 Embedding  NLP      .

 `konlpy` `sentence piece`    token  
`Word2vec`   Embedding    .

      .

1.  inference, training  tokenizer    
2. tokenization      ( tokenization)
3.    cover   ( OOV )


## Solution

    CNN   char-word   
`kor2vec`  .

- Embedding   : Skip-gram based embedding training
- Char-word Encoder   : [Yoon Kim's Character-Aware Neural Language Modeling](https://arxiv.org/abs/1508.06615)

## Quick Start

```shell
kor2vec train -c corpus/path -o output/model.kor2vec
```

### inference
```python

from kor2vec import Kor2Vec
kor2vec = Kor2Vec.load("../model/path")

kor2vec.embedding("    ")
>>> torch.tensor(5, 128) # embedding vector

kor2vec.embedding("   ", numpy=True)
>>> numpy.array(4, 128) # numpy embedding vector

input = kor2vec.to_seqs(["   ", "  "], seq_len=4)
kor2vec.forward(input)
>> torch.tensor([2, 4, 128])
```

### training

```python
from kor2vec import Kor2vec

kor2vec = Kor2Vec(embed_size=128)

kor2vec.train("../path/corpus", batch_size=128) # takes some time

kor2vec.save("../mode/path") # saving embedding
```

### with pytorch

```python

import torch.nn as nn
from kor2vec import Kor2vec

kor2vec = Kor2Vec.load("../model/path")
# or kor2vec = SejongVector()

lstm = nn.LSTM(128, 64, batch_first=True)
dense = nn.Linear(64, 1)

# Make tensor input
sentences = ["   ", "  "]

x = kor2vec.to_seqs(sentences, seq_len=10)
# >>> tensor(batch_size, seq_len, char_seq_len)

x = kor2vec(x) # tensor(batch_size, seq_len, 128)
_, (x, xc) = lstm(x) # tensor(batch_size, 64)
x = dense(x) # tensor(batch_size, 1)

```

## License

```
Copyright 2018 NAVER Corp.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```

## Author
 (Naver Clova AI intern) : codertimo@gmail.com

Owner

GitHub Events