https://github.com/carperai/code-pile
This repository contains all the code for collecting large scale amounts of code from GitHub.
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
1 of 5 committers (20.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (13.5%) to scientific vocabulary
Keywords
Repository
This repository contains all the code for collecting large scale amounts of code from GitHub.
Basic Info
Statistics
- Stars: 109
- Watchers: 8
- Forks: 31
- Open Issues: 22
- Releases: 0
Topics
Metadata Files
README.md
Code-Pile
This repository contains the processing scripts to scrape/process the code-pile dataset.
Table of Contents
- Project Description
- How to use the Code-Pile (todo)
- How to Contribute
- Additional Resources
Project Description
Check out The code pile proposal
The Code-Pile will be released similar to "the pile" as a folder of .jsonl.zst files, see lm-dataformat
How to use the Code-Pile
It's not finished, ask on discord
How to Contribute
Think about the most usefull Code-data for the next generation of textual Code Models.
The most valuable dataset properties (use your own judgment) are: 1. Open License 2. Data quality 3. Dataset size 4. Data variance/variety/nicheness 5. Ease of obtaining/processing
To add a new dataset, open a Issue under given dataset-request template. Gather all the related informations appropriate to it. Use the issue to track.
Check if there is existing Code or someone already working on it: See Additional Resources
- Eleuthers Pile V1 Repos
- Ask on Carper #code-pile
- Ask on Eleuther
- Consult the linked Spreadsheets below
Then implement it through the following steps:
- Fork this repo
- Use the
workingbranch - Read the shared classes in
datasets.pyandcodepile.py - Create mvp/example for your dataset
- Create a pull request
- Keep building the data-domain specific classes and repeat
Citation Placeholder:
@misc{Code-Pile,
author = {},
doi = {},
month = {},
title = {},
url = {https://github.com/CarperAI/Code-Pile},
version = {},
year = {2022}
}
Additional Resources
Closely related projects:
Previous work: * Codeparrot * ...
Owner
- Name: CarperAI
- Login: CarperAI
- Kind: organization
- Repositories: 15
- Profile: https://github.com/CarperAI
GitHub Events
Total
- Watch event: 5
- Fork event: 2
Last Year
- Watch event: 5
- Fork event: 2
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Reshinth Adithyan | 3****n | 6 |
| flowpoint | f****t | 6 |
| Nathan Cooper | n****1@w****u | 6 |
| duypv1 | v****1@v****o | 3 |
| Louis Castricato | w****p@g****m | 1 |
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 29
- Total pull requests: 24
- Average time to close issues: 4 months
- Average time to close pull requests: 9 days
- Total issue authors: 9
- Total pull request authors: 10
- Average comments per issue: 5.38
- Average comments per pull request: 3.08
- Merged pull requests: 18
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- ncoop57 (10)
- PhungVanDuy (5)
- dmahan93 (4)
- taisazero (2)
- reshinthadithyan (2)
- herbiebradley (1)
- Jehoshaph (1)
- aaronrmm (1)
- flowpoint (1)
Pull Request Authors
- PhungVanDuy (6)
- ncoop57 (4)
- flowpoint (4)
- vanga (3)
- jbaicoianu (2)
- herbiebradley (1)
- reshinthadithyan (1)
- faraday (1)
- Jehoshaph (1)
- LouisCastricato (1)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- actions/checkout v3 composite
- actions/setup-python v4 composite
- internetarchive *
- pydantic *