llama-factory-cangjie

https://github.com/codeshuttler/llama-factory-cangjie

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (11.5%) to scientific vocabulary

Last synced: 6 months ago · JSON representation ·

Repository

Basic Info

Host: GitHub
Owner: codeshuttler
License: apache-2.0
Language: Python
Default Branch: main
Size: 8.75 MB

Statistics

Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Releases: 0

Created 11 months ago · Last pushed 10 months ago

Metadata Files

Readme License Citation

Llama-Factory Fork: Training Code for "Translating to a Low-resource Language with Compiler Feedback: A Case Study on Cangjie"

This repository is a fork of Llama-Factory. It contains the training code used for the paper "Translating to a Low-resource Language with Compiler Feedback: A Case Study on Cangjie". The training parameters for pretrain, SFT, and KTO are provided in the hparams directory.

Getting Started

Installation

[!IMPORTANT] Installation is mandatory.

bash git clone --depth 1 https://github.com/codeshuttler/LLaMA-Factory-Cangjie.git cd LLaMA-Factory pip install -e ".[torch,metrics]"

Extra dependencies available: torch, torch-npu, metrics, deepspeed, liger-kernel, bitsandbytes, hqq, eetq, gptq, awq, aqlm, vllm, galore, badam, adam-mini, qwen, modelscope, openmind, quality

[!TIP] Use pip install --no-deps -e . to resolve package conflicts.

Data Preparation

For dataset details, refer to data/README.md. You can either use datasets from HuggingFace, ModelScope, or Modelers hub, or load a dataset from your local disk.

[!NOTE] Update data/custom_dataset_info.json to use your custom dataset.

Quickstart

After setting the path of the relevant dataset in data/custom_dataset_info.json, use the following commands to run Pretraining, Fine-tuning, and KTO for the qwen2-7b model:

bash llamafactory-cli train hparams/cangjie/cangjie-qwen2-7b/cangjie_pretrain.yaml llamafactory-cli train hparams/cangjie/cangjie-qwen2-7b/cangjie_sft.yaml llamafactory-cli train hparams/cangjie/cangjie-qwen2-7b/cangjie_lora_kto.yaml

For advanced usage, including distributed training, refer to examples/README.md.

[!TIP] Use llamafactory-cli help to show help information.

License

This repository is licensed under the Apache-2.0 License.

Follow the model licenses to use corresponding model weights: Baichuan 2 / BLOOM / Llama 3

Citation

If you find this work helpful, please cite it as:

bibtex ...

Acknowledgement

This repository benefits from PEFT, TRL, QLoRA, and FastChat. Thanks for their wonderful contributions.

Owner

Name: CodeWalker
Login: codeshuttler
Kind: user

Repositories: 1
Profile: https://github.com/codeshuttler

I am a code shutter.

Citation (CITATION.cff)

cff-version: 1.2.0
date-released: 2024-03
message: "If you use this software, please cite it as below."
authors:
- family-names: "Zheng"
  given-names: "Yaowei"
- family-names: "Zhang"
  given-names: "Richong"
- family-names: "Zhang"
  given-names: "Junhao"
- family-names: "Ye"
  given-names: "Yanhan"
- family-names: "Luo"
  given-names: "Zheyan"
- family-names: "Feng"
  given-names: "Zhangchi"
- family-names: "Ma"
  given-names: "Yongqiang"
title: "LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models"
url: "https://arxiv.org/abs/2403.13372"
preferred-citation:
  type: conference-paper
  conference:
    name: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)"
  authors:
    - family-names: "Zheng"
      given-names: "Yaowei"
    - family-names: "Zhang"
      given-names: "Richong"
    - family-names: "Zhang"
      given-names: "Junhao"
    - family-names: "Ye"
      given-names: "Yanhan"
    - family-names: "Luo"
      given-names: "Zheyan"
    - family-names: "Feng"
      given-names: "Zhangchi"
    - family-names: "Ma"
      given-names: "Yongqiang"
  title: "LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models"
  url: "https://arxiv.org/abs/2403.13372"
  year: 2024
  publisher: "Association for Computational Linguistics"
  address: "Bangkok, Thailand"

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science