Science Score: 67.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
✓DOI references
Found 6 DOI reference(s) in README -
✓Academic publication links
Links to: zenodo.org -
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.0%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
A HyFI plugin for Tokenizers
Basic Info
- Host: GitHub
- Owner: entelecheia
- License: mit
- Language: Python
- Default Branch: main
- Homepage: https://lexikanon.entelecheia.ai/
- Size: 6.42 MB
Statistics
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
- Releases: 26
Topics
Metadata Files
README.md
Lexikanon: A HyFI-based library for Tokenizers
A HyFI-based library for the creation, training, and utilization of tokenizers.
- Documentation: https://lexikanon.entelecheia.ai
- GitHub: https://github.com/entelecheia/lexikanon
- PyPI: https://pypi.org/project/lexikanon
Lexikanon is a high-performance Python library specifically engineered for the creation, training, and utilization of tokenizers, which are fundamental components in both natural language processing (NLP) and artificial intelligence (AI). Drawing its name from the Greek words λέξη (meaning "word") and κάνων (meaning "maker"), Lexikanon encapsulates its primary purpose of enabling users to develop robust tokenizers tailored for different languages and specific tasks. Built on the Hydra Fast Interface (HyFI) framework, Lexikanon stands as a HyFI-based library. This makes it seamlessly pluggable into any HyFI-oriented project, although it can also function as a standalone library.
Citation
tex
@software{lee_2023_8248118,
author = {Young Joon Lee},
title = {Lexikanon: A HyFI-based library for Tokenizers},
month = aug,
year = 2023,
publisher = {Zenodo},
version = {v0.6.2},
doi = {10.5281/zenodo.8248117},
url = {https://doi.org/10.5281/zenodo.8248117}
}
tex
@software{lee_2023_hyfi,
author = {Young Joon Lee},
title = {Lexikanon: A HyFI-based library for Tokenizers},
year = 2023,
publisher = {GitHub},
url = {https://github.com/entelecheia/lexikanon}
}
Changelog
See the CHANGELOG for more information.
Contributing
Contributions are welcome! Please see the contributing guidelines for more information.
License
This project is released under the MIT License.
Owner
- Name: Young Joon Lee
- Login: entelecheia
- Kind: user
- Location: Jeju, South Korea
- Company: Cheju Halla University
- Website: https://entelecheia.me
- Repositories: 12
- Profile: https://github.com/entelecheia
Professor at Cheju Halla University
Citation (CITATION.cff)
cff-version: 1.1.0
message: "If you use this software, please cite it as below."
authors:
- family-names: Lee
given-names: Young Joon
orcid: https://orcid.org/0000-0002-4349-5926
title: "Lexikanon: A HyFI-based library for Tokenizers"
version: v0.6.2
date-released: 2023-08-15
GitHub Events
Total
- Delete event: 7
- Issue comment event: 15
- Push event: 8
- Pull request review event: 11
- Pull request event: 16
- Create event: 9
Last Year
- Delete event: 7
- Issue comment event: 15
- Push event: 8
- Pull request review event: 11
- Pull request event: 16
- Create event: 9
Committers
Last synced: over 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Young Joon Lee | e****a@h****m | 187 |
| semantic-release | s****e | 26 |
| dependabot[bot] | 4****] | 26 |
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 1
- Total pull requests: 105
- Average time to close issues: 7 minutes
- Average time to close pull requests: 6 days
- Total issue authors: 1
- Total pull request authors: 2
- Average comments per issue: 0.0
- Average comments per pull request: 1.69
- Merged pull requests: 59
- Bot issues: 0
- Bot pull requests: 77
Past Year
- Issues: 0
- Pull requests: 14
- Average time to close issues: N/A
- Average time to close pull requests: 7 days
- Issue authors: 0
- Pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 1.93
- Merged pull requests: 11
- Bot issues: 0
- Bot pull requests: 14
Top Authors
Issue Authors
- entelecheia (1)
Pull Request Authors
- dependabot[bot] (91)
- entelecheia (27)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 76 last-month
- Total dependent packages: 3
- Total dependent repositories: 0
- Total versions: 26
- Total maintainers: 1
pypi.org: lexikanon
A Python Library for Tokenizers
- Homepage: https://lexikanon.entelecheia.ai
- Documentation: https://lexikanon.readthedocs.io/
- License: MIT
-
Latest release: 0.6.5
published almost 2 years ago