# Audio Captioning datasets for PyTorch Audio Captioning unofficial datasets source code for **AudioCaps** [[1]](#audiocaps), **Clotho** [[2]](#clotho), **MACS** [[3]](#macs), and **WavCaps** [[4]](#wavcaps), designed for PyTorch.

Installation

bash pip install aac-datasets

If you want to check if the package has been installed and the version, you can use this command: bash aac-datasets-info

Examples

Create Clotho dataset

```python from aac_datasets import Clotho

dataset = Clotho(root=".", download=True) item = dataset[0] audio, captions = item["audio"], item["captions"]

audio: Tensor of shape (nchannels=1, audiomax_size)

captions: list of str

```

Build PyTorch dataloader with Clotho

```python from torch.utils.data.dataloader import DataLoader from aacdatasets import Clotho from aacdatasets.utils.collate import BasicCollate

dataset = Clotho(root=".", download=True) dataloader = DataLoader(dataset, batchsize=4, collatefn=BasicCollate())

for batch in dataloader: # batch["audio"]: list of 4 tensors of shape (nchannels, audiosize) # batch["captions"]: list of 4 lists of str ... ```

Download datasets

To download a dataset, you can use download argument in dataset construction : python dataset = Clotho(root=".", subset="dev", download=True) However, if you want to download datasets from a script, you can also use the following command : bash aac-datasets-download --root "." clotho --subsets "dev"

Datasets information

aac-datasets package contains 4 different datasets :

| Dataset | Sampling
rate (kHz) | Estimated
size (GB) | Source | Subsets | |:---:|:---:|:---:|:---:|:---:| | AudioCaps | 32 | 43 | AudioSet | train
val
test
train_fixed | | Clotho | 44.1 | 53 | Freesound | dev
val
eval
dcase_aac_test
dcase_aac_analysis
dcase_t2a_audio
dcase_t2a_captions | | MACS | 48 | 13 | TAU Urban Acoustic Scenes 2019 | full | | WavCaps | 32 | 941 | AudioSet
BBC Sound Effects
Freesound
SoundBible | audioset
audioset_no_audiocaps_v1
bbc
freesound
freesound_no_clotho_v2
soundbible |

For Clotho, the dev subset should be used for training, val for validation and eval for testing.

Here is additional statistics of the train subsets from AudioCaps (v1), Clotho (v2.1), MACS and WavCaps:

| | AudioCaps/train | Clotho/dev | MACS/full | WavCaps/full | |:---:|:---:|:---:|:---:|:---:| | Nb audios | 49,838 | 3,840 | 3,930 | 403,050 | | Total audio duration (h) | 136.6<sup>1</sup> | 24.0 | 10.9 | 7563.3 | | Audio duration range (s) | 0.5-10 | 15-30 | 10 | 1-67,109 | | Nb captions per audio | 1 | 5 | 2-5 | 1 | | Nb captions | 49,838 | 19,195 | 17,275 | 403,050 | | Total nb words<sup>2</sup> | 402,482 | 217,362 | 160,006 | 3,161,823 | | Sentence size<sup>2</sup> | 2-52 | 8-20 | 5-40 | 2-38 | | Vocabulary<sup>2</sup> | 4724 | 4369 | 2721 | 24,600 | | Annotated by | Human | Human | Human | Machine | | Corrected by | Human | Human | None | None |

<sup>1</sup> This duration is estimated on the total duration of 46230/49838 files of 126.7h.

<sup>2</sup> The sentences are cleaned (lowercase+remove punctuation) and tokenized using the spacy tokenizer to count the words.

Requirements

This package has been developped for Ubuntu 20.04, and it is expected to work on most Linux-based distributions. It has been tested with Python versions 3.7 and 3.13.

Python packages

Python requirements are automatically installed when using pip on this repository. torch >= 1.10.1 torchaudio >= 0.10.1 py7zr >= 0.17.2 pyyaml >= 6.0 tqdm >= 4.64.0 huggingface-hub >= 0.15.1 numpy >= 1.21.2

External requirements (AudioCaps only)

The external requirements needed to download AudioCaps are ffmpeg and yt-dlp. ffmpeg can be installed on Ubuntu using sudo apt install ffmpeg and yt-dlp from the official repo.

You can also override their paths for AudioCaps: python from aac_datasets import AudioCaps dataset = AudioCaps( download=True, ffmpeg_path="/my/path/to/ffmpeg", ytdl_path="/my/path/to/ytdlp", )

Since YouTube prevents bots to download videos, you might want to use ytdlp_opts argument to use cookies to overcome failed downloads, e.g. AudioCaps(ytdlp_opts=["--cookies-from-browser", "firefox"]). See more information on the documentation of yt-dlp.

Additional information

Compatibility with audiocaps-download

If you want to use audiocaps-download 1.0 package to download AudioCaps (v1 only), you will have to respect the AudioCaps folder tree: python from audiocaps_download import Downloader root = "your/path/to/root" downloader = Downloader(root_path=f"{root}/AUDIOCAPS/audio_32000Hz/", n_jobs=16) downloader.download(format="wav")

Then disable audio download and set the correct audio format before init AudioCaps : python from aac_datasets import AudioCaps dataset = AudioCaps( root=root, subset="train", download=True, audio_format="wav", download_audio=False, # this will only download labels and metadata files )

References

AudioCaps

[1] C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generating captions for audios in the wild,” in NAACL-HLT, 2019. Available: https://aclanthology.org/N19-1011/

Clotho

[2] K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An Audio Captioning Dataset,” arXiv:1910.09387 [cs, eess], Oct. 2019, Available: http://arxiv.org/abs/1910.09387

MACS

[3] F. Font, A. Mesaros, D. P. W. Ellis, E. Fonseca, M. Fuentes, and B. Elizalde, Proceedings of the 6th Workshop on Detection and Classication of Acoustic Scenes and Events (DCASE 2021). Barcelona, Spain: Music Technology Group - Universitat Pompeu Fabra, Nov. 2021. Available: https://doi.org/10.5281/zenodo.5770113

WavCaps

[4] X. Mei et al., “WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research,” arXiv preprint arXiv:2303.17395, 2023, [Online]. Available: https://arxiv.org/pdf/2303.17395.pdf

Cite the aac-datasets package

If you use this software, please consider cite it as "Labbe, E. (2025). aac-datasets: Audio Captioning datasets for PyTorch.", or use the following BibTeX citation:

@software{ Labbe_aac_datasets_2025, author = {Labbé, Étienne}, license = {MIT}, month = {07}, title = {{aac-datasets}}, url = {https://github.com/Labbeti/aac-datasets/}, version = {0.7.0}, year = {2025} }

See also

• AudioCaps official data repository
• Clotho official data repository
• MACS official data repository
• WavCaps official data repository

Contact

Maintainer: - Étienne Labbé "Labbeti": labbeti.pub@gmail.com