https://github.com/aise-tudelft/forge-ds-intermediate
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (8.9%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: AISE-TUDelft
- Language: Python
- Default Branch: main
- Size: 61.8 MB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
The Heap
A contamination free code dataset for the evaluation and investigation of LLM behavior.
Layout
We give the code to reproduce the dataset in the code folder.
In the repositories folder, we give a list of all repositories we used to generate the dataset.
Running the code
Code Collection
- We start by scraping repositories from GitHub based on their creation date, license, and amount of stars, using repoextract.js_.
- We extract all files corresponding to the selected language from each repository, using extractfiles.py_.
Exact Deduplication
To run the exact deduplication we make use of unix (ubuntu) tools, the naming/availability may differ depending on the OS. 1. First we run hashentries.py_ To calculate and save to a text file all hashes belonging to our and other datasets. 2. We generate lists of unique hashes of our dataset, and the other dataset using exactdeduphashesself.py. 3. We merge two sets of hashes and record the duplicates using _exactdeduphashesother.py. 4. We flag duplicates in our dataset with respect to other datasets using _exactdedupdataset.py.
Near Deduplication
- We generate and save the LSH object containing all the minhashes of our exact deduplicated dataset, using lshcreation.py_.
- Using the LSH object, we perform near deduplication against other public datasets, using neardedup.py_.
Using the dataset
In order to have the most data available for each dataset, we do not filter duplicates from the dataset. Instead we add a boolean mask to The Heap that allows for filtering for unique files in each dataset.
Using the Datasets API, our dataset can be used as follows:
```python from datasets import load_dataset
dataset_name = 'redpajama' language = 'Python'
ds = loaddataset( "WizzF/Heap-Forge", f"{language}", split="train", numproc=16 )
ds = ds.filter(lambda x: not x[f'exactduplicates{datasetname}'] and not x[f'nearduplicates{datasetname}']) ```
Acknowledgements
We extended the collection of programming language extensions used for The Stack, in the file langsextension.json_ We added the EJS, Raku, Starlark, and WebAssembly languages.
Owner
- Name: AISE-TUDelft
- Login: AISE-TUDelft
- Kind: organization
- Repositories: 1
- Profile: https://github.com/AISE-TUDelft
GitHub Events
Total
- Watch event: 1
- Member event: 2
- Push event: 31
- Create event: 2
Last Year
- Watch event: 1
- Member event: 2
- Push event: 31
- Create event: 2
Issues and Pull Requests
Last synced: over 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0