Chinese-LLaMA-Alpaca-3

**** | English | /Docs | /Issues | /Discussions | /Arena

LLaMAAlpacaNLPLLaMAAlpaca

V2[Cui, Yang, and Yao] Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca

• LLaMA

- LLaMAAlpaca

• PCCPU/GPU
• transformers, llama.cpp, text-generation-webui, LlamaChat, LangChain, privateGPT
• 7BPlusPro13BPlusPro33BPlusPro

Alpaca-Plus-7BCPU

LLaMA-2&Alpaca-2 | LLaMA&Alpaca | VLE | MiniRBT | LERT | PERT | MacBERT | ELECTRA | XLNet | BERT | TextBrewer | TextPruner

[2024/04/30] Chinese-LLaMA-Alpaca-3 Llama-3Llama-3-Chinese-8BLlama-3-Chinese-8B-Instructhttps://github.com/ymcui/Chinese-LLaMA-Alpaca-3

[2024/03/27] SOTA!https://sota.jiqizhixin.com/project/chinese-llama-alpaca

[2023/08/14] Chinese-LLaMA-Alpaca-2 v2.0Chinese-LLaMA-2-13BChinese-Alpaca-2-13Bhttps://github.com/ymcui/Chinese-LLaMA-Alpaca-2

[2023/07/31] Chinese-LLaMA-Alpaca-2 v1.0https://github.com/ymcui/Chinese-LLaMA-Alpaca-2

[2023/07/19] v5.0: Alpaca-ProPlus-33B

[2023/07/19] LLaMA-2Alpaca-2

[2023/07/10] Beta

[2023/07/07] Chinese-LLaMA-AlpacaLLaMA&Alpaca7B

[2023/06/30] llama.cpp8K contexttransformers4K+ contextPR#705

[2023/06/16] v4.1: C-Eval

[2023/06/08] v4.0: LLaMA/Alpaca-33BprivateGPTC-Eval

| | | | ------------------------------------- | ------------------------------------------------------------ | | | LLaMAAlpaca | | | LoRALLaMA | | | | | | | | | LLaMAAlpaca | | FAQ | | | | |

FacebookLLaMALoRALLaMALLaMA/Alpaca LoRALLaMA

LLaMAAlpaca

| | LLaMA | Alpaca | | :-------------------- | ------------------------------------------------------ | ------------------------------------------------------------ | | | CLM | | | | | ChatGPT | | | | | | ^[3] | 49953 | 49954=49953+1pad token | | | | ^[1] | | | | | | | | | | llama.cpp | -p | -ins+ | | text-generation-webui | chat | --cpu | | LlamaChat | "LLaMA" | "Alpaca" | | HF | | --with_prompt | | web-demo | | Alpaca | | LangChain / privateGPT | | Alpaca | | | ^[2] | ProPlus |

[1] llama.cpp/LlamaChat/HF/web-demo/LangChain
[2] *
*[3] AlpacaLLaMApad tokenLLaMA/Alpaca**

ChatGPTAlpacaLLaMA AlpacaProPlus

| | | | ^[1] | ^[2] | LoRA^[3] | | :------------------------ | :------: | :------: | :--------------------------------------------------------: | :----------------: | :----------------------------------------------------------: | | Chinese-LLaMA-Plus-7B | | 120G | LLaMA-7B | 790M | [HF] [ModelScope] [Baidu] | | Chinese-LLaMA-Plus-13B | | 120G | LLaMA-13B | 1.0G | [HF] [ModelScope] [Baidu] | | Chinese-LLaMA-Plus-33B | | 120G | LLaMA-33B | 1.3G^[6] | [HF] [ModelScope] [Baidu] | | Chinese-Alpaca-Pro-7B | | 4.3M | LLaMA-7B &
LLaMA-Plus-7B^[4] | 1.1G | [HF] [ModelScope] [Baidu] | | Chinese-Alpaca-Pro-13B | | 4.3M | LLaMA-13B &
LLaMA-Plus-13B^[4] | 1.3G | [HF] [ModelScope] [Baidu] | | Chinese-Alpaca-Pro-33B | | 4.3M | LLaMA-33B &
LLaMA-Plus-33B^[4] | 2.1G | [HF] [ModelScope] [Baidu] |

[1] LLaMALLaMAPR
[2] LLaMA
[3] SHA256SHA256.md
[4] Alpaca-PlusLLaMA-Plus
[5] 30BFacebook33B
[6] FP16

Chinese-LLaMA-7B

chinese_llama_lora_7b/ - adapter_config.json # LoRA - adapter_model.bin # LoRA - special_tokens_map.json # special_tokens_map - tokenizer_config.json # tokenizer - tokenizer.model # tokenizer

****

| | | | | | LoRA | | :---------------- | :------: | :------: | :--------------------: | :----------------: | :----------------------------------------------------------: | | Chinese-LLaMA-7B | | 20G | LLaMA-7B | 770M | [HF] [ModelScope] [Baidu] | | Chinese-LLaMA-13B | | 20G | LLaMA-13B | 1.0G | [HF] [ModelScope] [Baidu] | | Chinese-LLaMA-33B | | 20G | LLaMA-33B | 2.7G | [HF] [ModelScope] [Baidu] | | Chinese-Alpaca-7B | | 2M | LLaMA-7B | 790M | [HF] [ModelScope] [Baidu] | | Chinese-Alpaca-13B | | 3M | LLaMA-13B | 1.1G | [HF] [ModelScope] [Baidu] | | Chinese-Alpaca-33B | | 4.3M | LLaMA-33B | 2.8G | [HF] [ModelScope] [Baidu] | | Chinese-Alpaca-Plus-7B | | 4M | LLaMA-7B &
LLaMA-Plus-7B | 1.1G | [HF] [ModelScope] [Baidu] | | Chinese-Alpaca-Plus-13B | | 4.3M | LLaMA-13B &
LLaMA-Plus-13B | 1.3G | [HF] [ModelScope] [Baidu] | | Chinese-Alpaca-Plus-33B | | 4.3M | LLaMA-33B &
LLaMA-Plus-33B | 2.1G | [HF] [ModelScope] [Baidu] |

transformers

Model HubtransformersPEFTLLaMAAlpaca LoRA.from_pretrained()

https://huggingface.co/hfl

LoRALLaMA

| | | | | :----------- | :--------------------------------------------------------- | :----------------------------------------------------------: | | **** | Colabnotebook | | | **** | | |

FP164-bit

| | 7B | 13B | 33B | 65B | | :------------------ | :----: | :-----: | :-----: | :-----: | | FP16 | 13 GB | 24 GB | 60 GB | 120 GB | | 8-bit | 7.8 GB | 14.9 GB | 32.4 GB | ~60 GB | | 4-bit | 3.9 GB | 7.8 GB | 17.2 GB | 38.5 GB |

GitHub Wiki

| | | | CPU | GPU | | | | | :----------------------------------------------------------- | -------------------------------------------- | :---: | :--: | :--: | :------: | :------: | :----------------------------------------------------------: | | llama.cpp | | | | | | | link | | Transformers | transformers | | | | | | link | | text-generation-webui | Web UI | | | | | | link | | LlamaChat | macOS | MacOS | | | | | link | | LangChain | LLM | | | | | | link | | privateGPT | LangChain | | | | | | link | | Colab Gradio Demo | ColabGradioWeb | | | | | | link | | API | OpenAI APIDemo | | | | | | link |

: LangChainLangChain

GitHub Wiki

promptAlpaca-7BAlpaca-13BAlpaca-33BAlpaca-Plus-7BAlpaca-Plus-13B

• examples
• Alpacahttp://llm-arena.ymcui.com

NLUmappingC-Eval12.3K52validtestAverage

| | Valid (zero-shot) | Valid (5-shot) | Test (zero-shot) | Test (5-shot) | | ----------------------- | :---------------: | :------------: | :--------------: | :-----------: | | Chinese-Alpaca-Plus-33B | 46.5 | 46.3 | 44.9 | 43.5 | | Chinese-Alpaca-33B | 43.3 | 42.6 | 41.6 | 40.4 | | Chinese-Alpaca-Plus-13B | 43.3 | 42.4 | 41.5 | 39.9 | | Chinese-Alpaca-Plus-7B | 36.7 | 32.9 | 36.4 | 32.3 | | Chinese-LLaMA-Plus-33B | 37.4 | 40.0 | 35.7 | 38.3 | | Chinese-LLaMA-33B | 34.9 | 38.4 | 34.6 | 39.5 | | Chinese-LLaMA-Plus-13B | 27.3 | 34.0 | 27.8 | 33.3 | | Chinese-LLaMA-Plus-7B | 27.3 | 28.3 | 26.9 | 28.4 |

C-Eval >>> GitHub Wiki

• LLaMAmerge_tokenizers.py
• transformersrun_clm.pyStanford Alpaca
• WikiWiki

GitHub Wiki

FAQ

FAQIssueFAQ

1 233B65B 3 4LLaMA 5 6Windows 7Chinese-LLaMA 13Bllama.cpp 8Chinese-Alpaca-Plus 9NLU 1033B30B 11SHA256

GitHub Wiki

-

• demo

https://arxiv.org/abs/2304.08177 @article{chinese-llama-alpaca, title={Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca}, author={Cui, Yiming and Yang, Ziqing and Yao, Xin}, journal={arXiv preprint arXiv:2304.08177}, url={https://arxiv.org/abs/2304.08177}, year={2023} }

| | | | | :----------------------------------------------------------- | :----------------------------- | :----: | | Chinese-LLaMA-Alpaca-2 | LLaMA-2Alpaca-2 | | | Visual-Chinese-LLaMA-Alpaca | LLaMA & Alpaca | |

| | | | | :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | | LLaMA by Facebook
Alpaca by Stanford
alpaca-lora by @tloen | llama.cpp by @ggerganov
LlamaChat by @alexrozanski
text-generation-webui by @oobabooga | pCLUE and MT data by @brightmart
oasst1 by OpenAssistant |

GitHub Issue

• FAQissue
• Issue
• issuestable-bot