Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/awslabs/vip-token-centric-compression

https://github.com/awslabs/vip-token-centric-compression

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (7.2%) to scientific vocabulary
Last synced: 10 months ago · JSON representation

Repository

Basic Info
  • Host: GitHub
  • Owner: awslabs
  • License: apache-2.0
  • Language: Python
  • Default Branch: main
  • Size: 150 KB
Statistics
  • Stars: 4
  • Watchers: 2
  • Forks: 0
  • Open Issues: 2
  • Releases: 0
Created over 2 years ago · Last pushed almost 2 years ago
Metadata Files
Readme Changelog Contributing License Code of conduct

README.md

Official Repo for VCC: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens

License

Paper link

VCC stands for VIP-Token Centric Compression

Dataset Preparation

All codes for dataset preparation are located at src/datasets_scripts

Finetuning

The codes simply download HuggingFace's Dataset or convert json files downloaded from official source to HuggingFace's Dataset Code for downstream datasets is very short and should be self-explanatory.

Pretraining

Use the following command to download wikipedia english dataset from HuggingFace python3 src/datasets_scripts/pretrain/wiki_en_dataset.py --output <output_path>

Then use the following command to segment or pack articles in wikipedia english dataset to examples of <sequence_length> length and stores examples to multiple jsonl files python3 src/datasets_scripts/pretrain/example_packing.py \ --output_folder <jsonl_files_output_folder> \ --data_file <wiki_en_output_path>/wikipedia.20220301.en/train/ \ --example_pack_length <sequence_length> \ --mp mp option will use all available cpu cores to preprocess the dataset.

Finally, use the following command to combine all jsonl files python3 src/datasets_scripts/pretrain/jsonl_to_arrow.py \ --jsonl_folder <jsonl_files_output_folder> \ --output_file <arrow_file_output_path> You can move some jsonl files from <jsonl_files_output_folder> to a different folder and use it as validation set.

Training using PyTorch Lightning

Both pretraining and finetuning are launched by main.py. All configuration including training pipeline, model, dataset, data collator, and optimizer are specified in a config file, such as cfgs/roberta/base-512/postnorm-16n.py python3 main.py --config cfgs/roberta/base-512/postnorm-16n.py

Owner

  • Name: Amazon Web Services - Labs
  • Login: awslabs
  • Kind: organization
  • Location: Seattle, WA

AWS Labs

GitHub Events

Total
Last Year

Issues and Pull Requests

Last synced: about 2 years ago

All Time
  • Total issues: 0
  • Total pull requests: 2
  • Average time to close issues: N/A
  • Average time to close pull requests: about 1 month
  • Total issue authors: 0
  • Total pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.5
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 2
Past Year
  • Issues: 0
  • Pull requests: 2
  • Average time to close issues: N/A
  • Average time to close pull requests: about 1 month
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.5
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 2
View more stats
Top Authors
Issue Authors
Pull Request Authors
  • dependabot[bot] (5)
Top Labels
Issue Labels
Pull Request Labels
dependencies (5)

Dependencies

requirements.txt pypi
  • PyWavelets ==1.4.1
  • apache_beam ==2.46.0
  • clip *
  • datasets ==2.5.0
  • einops ==0.6.0
  • einops_exts ==0.0.4
  • h5py ==3.8.0
  • ninja ==1.10.2
  • pytorch_lightning ==1.9.4
  • pytorch_wavelets ==1.3.0
  • resize_right ==0.0.2
  • sentencepiece ==0.1.98
  • spacy ==3.5.1
  • timm ==0.6.13
  • tqdm ==4.64.0
  • transformers ==4.20
  • wandb ==0.14.0