https://github.com/bao231/chinese-transformer-xl
Science Score: 10.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
○codemeta.json file
-
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (3.8%) to scientific vocabulary
Last synced: 10 months ago
·
JSON representation
Repository
Basic Info
- Host: GitHub
- Owner: bao231
- Default Branch: master
- Size: 1.12 MB
Statistics
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
- Releases: 0
Fork of BAAI-WuDao/Chinese-Transformer-XL
Created over 4 years ago
· Last pushed almost 5 years ago
https://github.com/bao231/Chinese-Transformer-XL/blob/master/
# Chinese-Transformer-XL
Under construction
""
Chinese-Transformer-XL[[]](https://gpt-3.aminer.cn/) [[]](http://dorc-model-team.ks3-cn-beijing.ksyun.com/ren-zhi/my-model/mp_rank_00_model_states.pt)
##
[WuDaoCorpus](https://data.baai.ac.cn/data-set-details/0c8dc71dd06ae75a10ca422fb49b0751)
WuDaoCorpus+133G131G38G303GB
##
[GPT-3](https://arxiv.org/abs/2005.14165)
[Transformer-XL](https://arxiv.org/abs/1901.02860) GPTTransformerGPT-3
2.7B32256032Transformer-XL29
##
[](https://www.zhihu.com)
10033[CPM](https://github.com/TsinghuaAI/CPM-Generate)
||(1-5)|(1-5)|(1-5)|(1-10)|
|---|---|---|---|---|
|CPM|2.66|2.47|2.36|4.32|
||3.44|3.25|3.21|5.97|
||3.80|3.61|3.67|6.85|
CPM""
##
`requirements.txt`pytorch
```shell
pip install -r requirements.txt
```
finetune[DeepSpeed](https://github.com/microsoft/DeepSpeed)
```shell
DS_BUILD_OPS=1 pip install deepspeed
```
##
[checkpoint](http://dorc-model-team.ks3-cn-beijing.ksyun.com/ren-zhi/my-model/mp_rank_00_model_states.pt)
```
.
txl-2.9B
mp_rank_00_model_states.pt
```
```shell
bash scripts/generate_text.sh ./txl-2.9B
```
## Finetune
finetuneDeepSpeed`scripts/ds_finetune_gpt_2.9B.sh``NUM_WORKERS``NUM_GPUS_PER_WORKER`
GPU`HOST_FILE_PATH`
hostfileDeepSpeed[OpenMPIhostfile](https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node)
finetune
```shell
bash scripts/ds_finetune_gpt_2.9B.sh ./txl-2.9B ./data.json
```
`./txl-2.9B`checkpoint`./data.json`finetune[jsonl](https://jsonlines.org/)
`{"prompt": .., "text": ...}`promptcontexttext
finetuneOOMGPU[scripts/ds_config_2.9B_finetune.json](scripts/ds_config_2.9B_finetune.json)`zero_optimization``"cpu_offload": true`[ZeRO-Offload](https://www.deepspeed.ai/tutorials/zero-offload/)
##
checkpoint[change_mp.py](change_mp.py)hceckpoint
```shell
python change_mp.py ./txl-2.9B 2
```
22finetuneMP_SIZE2./txl-2.9B_MP2checkpoint
##
Owner
- Login: bao231
- Kind: user
- Repositories: 84
- Profile: https://github.com/bao231