$D_{2}O$: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models

The code for ICLR 2025 paper: D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models.

📃 [Paper] • 💻 [Github] • 🤗 [Huggingface]

If you find our project helpful, please give us a star ⭐ on GitHub to stay updated.

Setup Environment

We recommend using Anaconda to create a new environment and install the required packages. You can create a new environment and install the required packages using the following commands: bash pip install -r requirements.txt conda create -n d2o_v2 python=3.10 conda activate d2o_v2 pip install --upgrade pip # enable PEP 660 support

Quick Step to Run the Code

You can run the inference code using the following command to run the Longbench sample: ```bash CUDAVISIBLEDEVICES=0 python runpredlongbenchsample.py --modelnameorpath meta-llama/Meta-Llama-3-8B \ --cachedir /yourhfhomepath \ --used2o True \ --modeltype llama3 \ --hhratio 0.1 \ --recentratio 0.1 \ --actionname d2o_0.2 \ --e True

`` -cachedirstores your model weights. -used2ospecifies the execution strategy name. -hhratiorefers to important tokens in our main paper. -recentratio` represents the proportion of the window closest to the generated token.

Then, evaluate the results: bash python eval_long_bench.py --model Meta-Llama-3-8B_d2o_0.2 --e

For tasks related to lm-evaluation-harness GitHub Repository,
we recommend using the latest version by running:

bash git clone https://github.com/EleutherAI/lm-evaluation-harness.git Then, follow the installation instructions provided in the repository and execute our algorithm accordingly.

Citation

bibtex @article{wan2024d2o, title={D2o: Dynamic discriminative operations for efficient generative inference of large language models}, author={Wan, Zhongwei and Wu, Xinjian and Zhang, Yu and Xin, Yi and Tao, Chaofan and Zhu, Zhihong and Wang, Xin and Luo, Siqi and Xiong, Jing and Zhang, Mi}, journal={arXiv preprint arXiv:2406.13035}, year={2024} }

or

bibtex @inproceedings{wan2025text, title={$$\backslash$text $\{$D$\}$ \_ $\{$2$\}$$\backslash$text $\{$O$\}$ $: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models}, author={Wan, Zhongwei and Wu, Xinjian and Zhang, Yu and Xin, Yi and Tao, Chaofan and Zhu, Zhihong and Wang, Xin and Luo, Siqi and Xiong, Jing and Wang, Longyue and others}, booktitle={The Thirteenth International Conference on Learning Representations} }