https://github.com/chrisdonahue/ddc_onset
Music onset detector from Dance Dance Convolution packaged as a lightweight PyTorch module
Science Score: 33.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
✓Committers with academic emails
1 of 1 committers (100.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (10.5%) to scientific vocabulary
Repository
Music onset detector from Dance Dance Convolution packaged as a lightweight PyTorch module
Basic Info
Statistics
- Stars: 38
- Watchers: 3
- Forks: 5
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Music onset detector from Dance Dance Convolution
This is a simple CPU- and GPU-compatible PyTorch port of the lightweight CNN-based music onset detector model from Dance Dance Convolution, originally referred to in that work as the "step placement" model.
I am resurfacing this model now as I suspect it may be useful for imbuing music audio generative models with fine-grained rhythmic control. Specifically, it can be used to extract time-varying onset salience functions from audio, which may be a reasonable musical analogy to the Canny edge detector used to enable pixel-wise control in ControlNet. A visual example:
Example usage
Installation: pip install git+https://github.com/chrisdonahue/ddc_onset
Example (additionally requires librosa and scipy).
```py import librosa
from ddconset import FRAMERATE, computeonsetsalience, find_peaks
audio is (5864816,) in [-1, 1]
audio, sr = librosa.load(librosa.example('fishin'), sr=44100, mono=True)
onset salience is (13301,) in [0, 1]
onsetsalience = computeonsetsalience(audio, sr) onsettimes = [frame / FRAMERATE for frame in findpeaks(onset_salience)] ```
Model behavior
Empirically, while this model was trained for a more specific task, its behavior is that of an onset detector (not a beat detector). In my experience, this model has solid recall both for drum and melodic onsets, though I have never attempted a formal comparison to other onset detectors.
The raw 100Hz onset salience function predicted by the model may be useful as a control signal for conditional generative models of music audio. Additionally, onsets can be discretized using simple a peak finding algorithm (blue dots in above image), and precision / recall can be refined by thresholding peaks (dashed blue line). In the context of controllable generative models, thresholding may be useful for improving generalization to user-specified onset controls.
Original training
This model was trained on data from the rhythm game Dance Dance Revolution. Specifically, it was trained as a binary classifer to predict whether or not each audio frame (100 per second) should correspond to the presence of a player action in the game (a "step").
Another input to the model during training was a user-specified "difficulty" (1-5), corresponding to difficulties assigned by level designers in the original game. The hope at the time was that the model could learn to adjust predictions based on a user's skill level. This never really worked that well and isn't particularly relevant for onset detection. Hence, this input is set to the highest difficulty by default, but the option is there if you want to play with this setting.
This model was originally implemented and trained in TensorFlow. I ported that ugly implementation to a clean PyTorch one a few years back as part of the process of building Beat Sage.
Citing this work
If you end up using this for anything, please cite the original DDC paper.
@inproceedings{donahue2017ddc,
title={{Dance Dance Convolution}},
author={Donahue, Chris and Lipton, Zachary C and McAuley, Julian},
booktitle={ICML},
year={2017},
}
Owner
- Name: Chris Donahue
- Login: chrisdonahue
- Kind: user
- Location: Pittsburgh
- Website: chrisdonahue.com
- Twitter: chrisdonahuey
- Repositories: 55
- Profile: https://github.com/chrisdonahue
Assistant professor @ CMU CSD. Part-time research scientist at Google Magenta. Machine learning for music, and creative interaction.
GitHub Events
Total
- Watch event: 6
- Fork event: 1
Last Year
- Watch event: 6
- Fork event: 1
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Chris Donahue | c****e@c****u | 15 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 1
- Total pull requests: 0
- Average time to close issues: 10 days
- Average time to close pull requests: N/A
- Total issue authors: 1
- Total pull request authors: 0
- Average comments per issue: 4.0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- stet-stet (1)