miditok

MIDI / symbolic music tokenizers for Deep Learning models 🎶

https://github.com/natooz/miditok

Science Score: 77.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
✓
DOI references
Found 3 DOI reference(s) in README
✓
Academic publication links
Links to: arxiv.org, springer.com, acm.org
✓
Committers with academic emails
1 of 11 committers (9.1%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (13.4%) to scientific vocabulary

Keywords

deep-learning generative-model machine-learning midi music music-generation music-information-retrieval

Last synced: 6 months ago · JSON representation ·

Repository

MIDI / symbolic music tokenizers for Deep Learning models 🎶

Basic Info

Host: GitHub
Owner: Natooz
License: mit
Language: Python
Default Branch: main
Homepage: https://miditok.readthedocs.io/
Size: 5.74 MB

Statistics

Stars: 792
Watchers: 11
Forks: 95
Open Issues: 2
Releases: 0

Topics

deep-learning generative-model machine-learning midi music music-generation music-information-retrieval

Created over 4 years ago · Last pushed 6 months ago

Metadata Files

Readme Contributing License Citation

MidiTok

Python package to tokenize music files, introduced at the ISMIR 2021 LBDs.

MidiTok Logo

MidiTok can tokenize MIDI and abc files, i.e. convert them into sequences of tokens ready to be fed to models such as Transformer, for any generation, transcription or MIR task. MidiTok features most known music tokenizations (e.g. REMI, Compound Word...), and is built around the idea that they all share common parameters and methods. Tokenizers can be trained with Byte Pair Encoding (BPE), Unigram and WordPiece, and it offers data augmentation methods.

MidiTok is integrated with the Hugging Face Hub 🤗! Don't hesitate to share your models to the community!

Documentation: miditok.readthedocs.com

Install

shell pip install miditok MidiTok uses Symusic to read and write MIDI and abc files, and BPE/Unigram is backed by Hugging Face 🤗tokenizers for superfast encoding.

Usage example

Tokenizing and detokenzing can be done by calling the tokenizer:

```python from miditok import REMI, TokenizerConfig from symusic import Score

Creating a multitrack tokenizer, read the doc to explore all the parameters

config = TokenizerConfig(numvelocities=16, usechords=True, use_programs=True) tokenizer = REMI(config)

Loads a midi, converts to tokens, and back to a MIDI

midi = Score("path/to/yourmidi.mid") tokens = tokenizer(midi) # calling the tokenizer will automatically detect MIDIs, paths and tokens convertedback_midi = tokenizer(tokens) # PyTorch, Tensorflow and Numpy tensors are supported ```

Here is a complete yet concise example of how you can use MidiTok to train any PyTorch model. And here is a simple notebook example showing how to use Hugging Face models to generate music, with MidiTok taking care of tokenizing music files.

```python from miditok import REMI, TokenizerConfig from miditok.pytorchdata import DatasetMIDI, DataCollator from miditok.utils import splitfilesfortraining from torch.utils.data import DataLoader from pathlib import Path

Creating a multitrack tokenizer, read the doc to explore all the parameters

config = TokenizerConfig(numvelocities=16, usechords=True, use_programs=True) tokenizer = REMI(config)

Train the tokenizer with Byte Pair Encoding (BPE)

filespaths = list(Path("path", "to", "midis").glob("*/.mid")) tokenizer.train(vocabsize=30000, filespaths=filespaths) tokenizer.save(Path("path", "to", "save", "tokenizer.json"))

And pushing it to the Hugging Face hub (you can download it back with .from_pretrained)

tokenizer.pushtohub("username/model-name", private=True, token="yourhftoken")

Split MIDIs into smaller chunks for training

datasetchunksdir = Path("path", "to", "midichunks") splitfilesfortraining( filespaths=filespaths, tokenizer=tokenizer, savedir=datasetchunksdir, maxseq_len=1024, )

Create a Dataset, a DataLoader and a collator to train a model

dataset = DatasetMIDI( filespaths=list(datasetchunksdir.glob("*/.mid")), tokenizer=tokenizer, maxseqlen=1024, bostokenid=tokenizer["BOSNone"], eostokenid=tokenizer["EOSNone"], ) collator = DataCollator(tokenizer.padtokenid, copyinputsaslabels=True) dataloader = DataLoader(dataset, batchsize=64, collatefn=collator)

Iterate over the dataloader to train a model

for batch in dataloader: print("Train your model on this batch...") ```

Tokenizations

MidiTok implements the tokenizations: (links to original papers) * REMI * REMI+ * MIDI-Like * TSD * Structured * CPWord * Octuple * MuMIDI * MMM * PerTok

You can find short presentations in the documentation.

Contributions

Contributions are gratefully welcomed, feel free to open an issue or send a PR if you want to add a tokenization or speed up the code. You can read the contribution guide for details.

Todos

Support music-xml files;
no_duration_drums option, discarding duration tokens for drum notes;
Control Change messages;
Speed-up global/track events parsing with Rust or C++ bindings.

Citation

If you use MidiTok for your research, a citation in your manuscript would be gladly appreciated. ❤️

[MidiTok paper] [MidiTok original ISMIR publication] bibtex @inproceedings{miditok2021, title={{MidiTok}: A Python package for {MIDI} file tokenization}, author={Fradet, Nathan and Briot, Jean-Pierre and Chhel, Fabien and El Fallah Seghrouchni, Amal and Gutowski, Nicolas}, booktitle={Extended Abstracts for the Late-Breaking Demo Session of the 22nd International Society for Music Information Retrieval Conference}, year={2021}, url={https://archives.ismir.net/ismir2021/latebreaking/000005.pdf}, }

The BibTeX citations of all tokenizations can be found in the documentation

Acknowledgments

@Natooz thanks its employers who allowed him to develop this project, by chronological order Aubay, the LIP6 (Sorbonne University), and the Metacreation Lab (Simon Fraser University).

All Thanks To Our Contributors

Owner

Name: Nathan Fradet
Login: Natooz
Kind: user
Location: France
Company: @Metacreation-Lab

Website: nathanfradet.com
Twitter: NathanFradet
Repositories: 10
Profile: https://github.com/Natooz

AI Researcher working on Music Generation

Citation (CITATION.cff)

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Fradet"
  given-names: "Nathan"
  orcid: "https://orcid.org/0000-0003-4729-570X"
- family-names: "Briot"
  given-names: "Jean-Pierre"
  orcid: "https://orcid.org/0000-0003-1621-6335"
- family-names: "Chhel"
  given-names: "Fabien"
  orcid: "https://orcid.org/0000-0003-2224-8296"
- family-names: "El Fallah Seghrouchni"
  given-names: "Amal"
  orcid: "https://orcid.org/0000-0002-8390-8780"
- family-names: "Gutowski"
  given-names: "Nicolas"
  orcid: "https://orcid.org/0000-0002-5765-9901"
title: "MidiTok: A Python package for MIDI file tokenization"
license: MIT
date-released: 2021-11-07
url: "https://github.com/Natooz/MidiTok"
repository-code: "https://github.com/Natooz/MidiTok"

GitHub Events

Total

Create event: 18
Release event: 3
Issues event: 30
Watch event: 115
Delete event: 15
Issue comment event: 104
Push event: 27
Pull request review event: 7
Pull request review comment event: 8
Pull request event: 50
Fork event: 19

Last Year

Create event: 18
Release event: 3
Issues event: 30
Watch event: 115
Delete event: 15
Issue comment event: 104
Push event: 27
Pull request review event: 7
Pull request review comment event: 8
Pull request event: 50
Fork event: 19

Committers

Last synced: almost 3 years ago

All Time

Total Commits: 286
Total Committers: 11
Avg Commits per committer: 26.0
Development Distribution Score (DDS): 0.35

Top Committers

Name	Email	Commits
Nat	n**t@i**m	186
Nathan Fradet	5**z@u**m	58
atsukoba	a**a@s**p	17
Nathan Fradet		6
Ilya Borovik	i**7@g**m	6
Alex	a**1@h**m	6
Megha Sharma	5**4@u**m	3
Kian-Meng Ang	k**g@c**g	1
nturusin	o**e@g**m	1
dinhviettoanle	l**n@g**m	1
adamoudad	a**d@g**m	1

Committer Domains (Top 20 + Academic)

cpan.org: 1 sfc.keio.ac.jp: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 99
Total pull requests: 148
Average time to close issues: 27 days
Average time to close pull requests: 3 days
Total issue authors: 66
Total pull request authors: 39
Average comments per issue: 5.98
Average comments per pull request: 1.34
Merged pull requests: 131
Bot issues: 0
Bot pull requests: 19

Past Year

Issues: 16
Pull requests: 44
Average time to close issues: about 1 month
Average time to close pull requests: about 21 hours
Issue authors: 14
Pull request authors: 11
Average comments per issue: 3.94
Average comments per pull request: 0.95
Merged pull requests: 33
Bot issues: 0
Bot pull requests: 15

View more stats

Top Authors

Issue Authors

oiabtt (5)
Kapitan11 (4)
webpolis (3)
asigalov61 (3)
leleogere (3)
Kinyugo (3)
caenopy (3)
juancopi81 (3)
kroll-software (3)
parneyw (3)
pstrepetov (2)
Fansesi (2)
JLenzy (2)
ms3744 (2)
MHZ9825 (2)

Pull Request Authors

Natooz (105)
dependabot[bot] (28)
JLenzy (9)
ilya16 (7)
Mintas (4)
shenranwang (3)
adricl (2)
helloWorld199 (2)
gfggithubleet (2)
ms3744 (2)
sunsetsobserver (2)
emmanuel-ferdman (2)
scottclowe (2)
briane412 (2)
THEFZNKHAN (2)

Top Labels

Issue Labels

stale (43) bug (17) enhancement (11) help wanted (8) question (8) good first issue (4) documentation (2) invalid (1)

Pull Request Labels

dependencies (28) enhancement (11) github_actions (6) documentation (4) bug (4)

Packages

Total packages: 1
Total downloads:
- pypi 7,052 last-month

Total dependent packages: 2
Total dependent repositories: 2
Total versions: 67
Total maintainers: 1

pypi.org: miditok

MIDI / symbolic music tokenizers for Deep Learning models.

Homepage: https://github.com/Natooz/MidiTok
Documentation: https://miditok.readthedocs.io
License: MIT License Copyright (c) 2021 Nathan Fradet Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Latest release: 3.0.6
published 8 months ago

Versions: 67
Dependent Packages: 2
Dependent Repositories: 2
Downloads: 7,052 Last month

Rankings

Stargazers count: 2.9%

Dependent packages count: 3.2%

Forks count: 5.6%

Average: 5.9%

Downloads: 6.1%

Dependent repos count: 11.8%

Maintainers (1)

Natooz

Last synced: 6 months ago

Dependencies

.github/workflows/close-stale-issues.yml actions

actions/stale v5.1.1 composite

.github/workflows/pytest.yml actions

actions/checkout v3 composite
actions/setup-python v4 composite
codecov/codecov-action v3.1.0 composite

.github/workflows/python-publish.yml actions

actions/checkout v3 composite
actions/setup-python v3 composite
pypa/gh-action-pypi-publish 27b31702a0e7fc50959f5ad993c78deac1bdfc29 composite

docs/requirements.txt pypi

huggingface_hub *
matplotlib *
miditoolkit >=0.1.16
numpy >=1.19,<1.24
scipy *
sphinx-rtd-theme *
tokenizers >=0.13.2
torch *
tqdm >=4.64.0

requirements.txt pypi

huggingface_hub >=0.16.4
matplotlib *
miditoolkit >=0.1.16
numpy >=1.19,<1.24
scipy *
tokenizers >=0.13.2
tqdm >=4.64.0

setup.py pypi

huggingface_hub >=0.16.4
matplotlib *
miditoolkit >=0.1.16
numpy >=1.19,<1.24
scipy *
tokenizers >=0.13.0
tqdm *

miditok

Science Score: 77.0%

Keywords

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

MidiTok

Install

Usage example

Creating a multitrack tokenizer, read the doc to explore all the parameters

Loads a midi, converts to tokens, and back to a MIDI

Creating a multitrack tokenizer, read the doc to explore all the parameters

Train the tokenizer with Byte Pair Encoding (BPE)

And pushing it to the Hugging Face hub (you can download it back with .from_pretrained)

Split MIDIs into smaller chunks for training

Create a Dataset, a DataLoader and a collator to train a model

Iterate over the dataloader to train a model

Tokenizations

Contributions

Todos

Citation

Acknowledgments

All Thanks To Our Contributors

Owner

Citation (CITATION.cff)

GitHub Events

Total

Last Year

Committers

All Time

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: miditok

Rankings

Maintainers (1)

Dependencies