https://github.com/carperai/squeakily
A library for squeakily cleaning and filtering language datasets.
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
3 of 5 committers (60.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (10.9%) to scientific vocabulary
Keywords from Contributors
Repository
A library for squeakily cleaning and filtering language datasets.
Basic Info
- Host: GitHub
- Owner: CarperAI
- License: apache-2.0
- Language: Jupyter Notebook
- Default Branch: main
- Homepage: https://carperai.github.io/squeakily/
- Size: 1.29 MB
Statistics
- Stars: 47
- Watchers: 3
- Forks: 10
- Open Issues: 4
- Releases: 0
Metadata Files
README.md
squeakily
This repository is heavily inspired by BigScience’s ROOTs project and EleutherAI’s The Pile.
The overall pipeline is as follows:
mermaid
flowchart LR
A(Defining <br/>Datasources) --> B(Defining Filters <br/>per Datasource)
B --> C(Defining Cleaners <br/>per Datasource)
In this library, we define filtering as data instances being removed from the dataset based on some criteria and cleaning as data instances being modified in some way.
Install
sh
pip install squeakily
How to use
Using the API
First, we need to define a datasource. squeakily accepts any Dataset
object from the HuggingFace
Datasets library. For
example, we can use the
wikitext dataset:
``` python from datasets import load_dataset
ds = load_dataset("wikitext", "wikitext-103-v1", split="train[:1%]") ```
We simply need to wrap the Dataset object in a dictionary, with the
key being the name of the datasource and the value being the Dataset
object, the filter and cleaners. For example:
``` python from squeakily.filter import checkcharrepetition, checkflaggedwords from squeakily.clean import removeemptylines, normalize_whitespace
datasources = [ { "dataset": ds, "name": "wikitext", "columns": ["text"], "filters": [checkcharrepetition, checkflaggedwords], "cleaners": [removeemptylines, normalize_whitespace], }, # ... ] ```
Finally, we can apply the filters and cleaners to the datasouces using a
Pipeline
object:
``` python from squeakily.core import Pipeline
pipeline = Pipeline(datasources) pipeline.run() ```
[11/16/22 04:32:57] INFO Running datasource: wikitext core.py:41
INFO Running filter: check_char_repetition on text core.py:54
INFO Running filter: check_flagged_words on text core.py:54
INFO Running cleaner: remove_empty_lines on text core.py:57
[11/16/22 04:32:59] INFO Running cleaner: normalize_whitespace on text core.py:57
If you need to run a filter or cleaner at the dataset level rather than
the example level, you can pass global_filters or global_cleaners to
the
Pipeline.run
function. For example:
``` python from squeakily.filter import minhash_dedup
pipeline.run(globalfilters=[minhashdedup]) ```
Additionally, you can run the pipeline in a dry run mode by passing
dry_run=True to the run function. This will make no modifications to
the datasets’ documents, but will add additional columns to the datasets
with the results of the filters and cleaners. For example, if you if you
ran the pipeline with the
check_char_repetition
filter, you would get a new column called
check_char_repetition
with a float value between 0 and 1 indicating the percentage of
characters that are repeated in the document.
``` python
::: {.cell}
{.python .cell-code}
pipeline = Pipeline(datasources)
pipeline.run(dry_run=True)
pipeline.datasources[0]["dataset"].features
:::
Owner
- Name: CarperAI
- Login: CarperAI
- Kind: organization
- Repositories: 15
- Profile: https://github.com/CarperAI
GitHub Events
Total
- Watch event: 2
- Fork event: 1
Last Year
- Watch event: 2
- Fork event: 1
Committers
Last synced: over 3 years ago
All Time
- Total Commits: 33
- Total Committers: 5
- Avg Commits per committer: 6.6
- Development Distribution Score (DDS): 0.424
Top Committers
| Name | Commits | |
|---|---|---|
| ncoop57 | n****1@w****u | 19 |
| ncoop57 | n****1@e****u | 5 |
| nathan | n****n@i****l | 4 |
| Erfan Al-Hossami | e****a@u****u | 4 |
| nathan | n****n@c****r | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 11 months ago
All Time
- Total issues: 4
- Total pull requests: 8
- Average time to close issues: 8 days
- Average time to close pull requests: 2 days
- Total issue authors: 1
- Total pull request authors: 4
- Average comments per issue: 0.0
- Average comments per pull request: 0.88
- Merged pull requests: 6
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- ncoop57 (4)
Pull Request Authors
- ncoop57 (3)
- simugrad (2)
- taisazero (2)
- reshinthadithyan (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 13 last-month
- Total dependent packages: 0
- Total dependent repositories: 1
- Total versions: 2
- Total maintainers: 1
pypi.org: squeakily
A library for squeakily cleaning and filtering language datasets.
- Homepage: https://github.com/CarperAI/squeakily
- Documentation: https://squeakily.readthedocs.io/
- License: Apache Software License 2.0
-
Latest release: 0.0.2
published over 3 years ago
Rankings
Maintainers (1)
Dependencies
- fastai/workflows/quarto-ghp master composite
- fastai/workflows/nbdev-ci master composite