https://github.com/amazon-science/buggy-code-completion
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (8.5%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: amazon-science
- License: apache-2.0
- Language: Python
- Default Branch: main
- Size: 124 KB
Statistics
- Stars: 4
- Watchers: 3
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Large Language Models of Code Fail at Completing Code with Potential Bugs
Overview
This folder contains implementations and scripts for the NeurIPS 2023 paper "Large Language Models of Code Fail at Completing Code with Potential Bugs".
Usage
Installation
pip install transformers==4.14.1
pip install accelerate
pip install fire
pip install code-tokenize
Datasets
See instructions for buggy-HumanEval and buggy-FixEval datasets under the data folder.
Source codes
All source codes are stored in the src/ folder
For the baseline, download the Realit package (NBFModel) and put under the src/
Run experiments
To run completion, use
python completor.py --dataset humaneval --mode buggy --model_name codegen-2B-mono --method completion --batch_size 15 --large 1
or use the wandb:
- modify the configurations in sweep_config.yml and copy sweep id into run.sh
- ./run.sh
Here,
- dataset: the name of dataset, either humameval or fixeval
- large: version of dataset, 1 for large and 0 for small
- mode: setting of the partial code, clean or buggy
- model_name: name of code language models, [codegen-2B-mono, codegen-350M-mono, incoder-6B, incoder-1B]
- method: [completion, removal, comp_fix, infill_line] for four methods in our paper
Evaluation
To evaluate, use the scripts/eval.sh script.
For example, here is the script to evaluate the pass@k of buggy completion on large buggy-HumanEval with codegen-2B-mono using removal-completion method:
sh scripts/eval.sh 0 0-1903 buggy codegen-2B-mono removal
Get results
Results will be saved under the directory results/evals/
Owner
- Name: Amazon Science
- Login: amazon-science
- Kind: organization
- Website: https://amazon.science
- Twitter: AmazonScience
- Repositories: 80
- Profile: https://github.com/amazon-science
GitHub Events
Total
- Watch event: 1
- Fork event: 1
Last Year
- Watch event: 1
- Fork event: 1
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0