https://github.com/alixunxing/cluecorpus2020

Large-scale Pre-training Corpus for Chinese 100G 中文预训练语料

https://github.com/alixunxing/cluecorpus2020

Repository

Large-scale Pre-training Corpus for Chinese 100G 中文预训练语料

https://github.com/alixunxing/CLUECorpus2020/blob/master/

# CLUECorpus2020

## 

Common Crawl100GB 

** https://arxiv.org/pdf/2003.01355**

![./pics/corpus.png](./pics/corpus.png)

#### 
1. 
2. NLP

## 

Google

| Token Type | Google | CLUE |
| :----:| :----: | :----: |
| Simplified Chinese | 11378 | 5689 |
| Traditional Chinese | 3264 |  |
| English | 3529 | 1320 |
| Japanese | 573 |  |
| Korean | 84 |  |
| Emoji | 56 |  |
| Numbers | 1179 | 140 |
| Special Tokens | 106 | 106 |
| Other Tokens | 959 | 766 |
| Total | 21128 | 8021 |

## 

BERT-base

| Model        | Vocab  | Data        | Steps | AFQMC  | TNEWS'  | IFLYTEK'  | CMNLI  |  AVG   |
| :----:| :----: | :----: | :----: |:----: |:----: |:----: |:----: |:----: |
| BERT-base    | Google | Wiki (1 GB) | 125K  | 69.93% | 54.77%  | 57.54%    | 75.64% | 64.47% |
| BERT-base    | Google | C5 (1 GB)   | 125K  | 69.63% | 55.72%  | 58.87%    | 75.75% | 64.99% |
| BERT-base    | CLUE   | C5 (1 GB)   | 125K  | 69.00% | 55.04%  | 59.07%    | 75.84% | 64.74% |
| BERT-base mm | Google | C5 (1 GB)   | 125K  | 69.57% | 55.17%  | 59.69%    | 75.86% | 65.07% |
| BERT-base    | Google | C5 (1 GB)   | 375K  | 69.85% | 55.97%  | 59.62%    | 76.41% | 65.46% |
| BERT-base    | CLUE   | C5 (1 GB)   | 375K  | 69.93% | 56.38%  | 59.35%    | 76.58% | 65.56% |
| BERT-base    | Google | C5 (3 GB)   | 375K  | 70.22% | 56.41%  | 59.58%    | 76.70% | 65.73% |
| BERT-base    | CLUE   | C5 (3 GB)   | 375K  | 69.49% | 55.97%  | 60.12%    | 77.66% | 65.81% |

CLUEPretrainedModels

## 




: CLUEbenchmark@163.comCLUECorpus2020 100G

# CLUECorpusSmall14G

14G4000txt50nlp_chinese_corpus

4M

14G

1 news2016zh_corpus: 8G2000  :mzlk

2- webText2019zh_corpus3G3G900 :qvlq

3- wiki2019zh_corpus1.1G300  :rja4

4- comments2019zh_corpus2.3G784547227ChineseNLPCorpus  :5kwk

## 

issue(QQ:836811304)

 CLUEbenchmark@163.com

Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC)


## 

    @article{CLUECorpus2020,
      title={CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model},
      author={Liang Xu and Xuanwei Zhang and Qianqian Dong},
      journal={ArXiv},
      year={2020},
      volume={abs/2003.01355}
    }

## 

CLUE

****

Owner

GitHub Events