IndicSUPERB

IndicSUPERB is a robust benchmark consisting of 6 speech language understanding (SLU) tasks across 12 Indian languages. The tasks include automatic speech recognition, automatic speaker verification, speech idntification, query by example and keyword spotting. The IndicSUPERB also encompasses Kathbath dataset which has 1684 hours of labelled speech data across 12 Indian Languages.

Read more in our paper - IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages

Kathbath Dataset Details

| | bengali| gujarati| hindi |kannada| malayalam| marathi| odia| punjabi| sanskrit| tamil| telugu| urdu| |-|-|-|-|-|-|-|-|-|-|-|-|-| Data duration (hours) |115.8 |129.3 |150.2 |65.8 |147.3 |185.2 |111.6 |136.9 |115.5 |185.1 |154.9 |86.7| No. of male speakers | 18 | 44 | 58 | 53 | 12 | 82 | 10 | 65 | 95 | 116 | 53 |36 | No. of female speakers | 28 |35 |63 |26 |20 |61 |32 |77 |110 |42| 51|31| Vocabulary (no. of unique words)| 6k | 109k | 54k | 181k | 268k | 132k | 94k | 56k | 298k | 171k | 147k | 44k |

Downloads

The dataset can be downloaded from the links given below.

Download Links (Clean split):

• Train: 85GB
• Valid: 3GB
• Test Known: 2GB
• Test Unknown: 2GB

Transcripts: clean

Download Links (Noisy split):

• Test Known: 2GB
• Test Unknown: 1.4GB

Transcripts: noisy

Audio Dataset Format

• The audio files for each language are present in separate folders.
• The speaker and gender information are present in the name of the audio file.
• The audio files are stored in m4a format. For resampling, please check the sample code here

Folder Structure of audios after extraction

``` Audio Data data ├── bengali │ ├── │ │ ├── 844424931537866-594-f.m4a │ │ ├── 844424931029859-973-f.m4a │ │ ├── ... ├── gujarati ├── ...

Transcripts data ├── bengali │ ├── │ │ ├── transcription_n2w.txt ├── gujarati ├── ... ```

IndicSUPERB Tasks

In this section, we describe the tasks included in IndicSUPERB - automatic speech recognition, automatic speaker verification, speech idntification, query by example and keyword spotting.

Download the Pretrained checkpoint

You can download the IndicWav2Vec pretrained checkpoint from here.

Data Preprocessing

To prepare data for the downstream tasks, the following 3 scripts need to be run to preprocess the raw downloaded data. You can download the data from here.

1. Convert m4a files to wav format python utilities/structure.py \ <dataset_root_path>/kb_data_clean_m4a \ <dataset_root_path>/kb_data_clean_wav \ <lang>

2. Preprocess data for ASR task python utilities/preprocess_asr.py <dataset_root_path>/kb_data_clean_wav

Language Identification

Language Identification, LID is the task of identifying the language of an audio clip. In other words, the task is to take the raw audio as input and classify it into one of the given n languages (n = 12 in this case).

Training -

python s3prl/run_downstream.py -n lid_indicw2v -m train -u wav2vec2_local -d indic_lid -k <pretraining_checkpoint>

Evalutation -

overridebs="config.downstream_expert.datarc.eval_batch_size=1,,config.downstream_expert.datarc.file_path=<data_root>/kb_data_clean_wav" python s3prl/run_downstream.py -m evaluate -e <trained_checkpoint> -t test_known -o $overridebs python s3prl/run_downstream.py -m evaluate -e <trained_checkpoint> -t test_unknown -o $overridebs

Speaker Identification

Speaker Identification, SID classifies each utterance for its speaker identity as a multi-class classification, where speakers are in the same predefined set for both training and testing.

Training -

python s3prl/run_downstream.py -n <exp_name> -m train -u wav2vec2_local -d indic_sid -o "config.downstream_expert.datarc.file_path=<data_root>/kb_data_clean_wav/<lang>" -k <pretraining_checkpoint>

Evaluation -

overridebs="config.downstream_expert.datarc.eval_batch_size=1,,config.downstream_expert.datarc.file_path=<data_root>/kb_data_clean_wav/<lang>" python s3prl/run_downstream.py -m evaluate -e <trained_checkpoint> -t test_known -o $overridebs

Multilingual Speaker Identification

Training -

python s3prl/run_downstream.py -n <exp_name> -m train -u wav2vec2_local -d indic_sid_multi -k <pretraining_checkpoint>

Evaluate -

python s3prl/run_downstream.py -m evaluate -e <trained_checkpoint> -t test_known python s3prl/run_downstream.py -m evaluate -e <trained_checkpoint> -t test_unknown

Automatic Speaker Verification

Automatic Speaker Verification, ASV verifies whether the speakers of a pair of utterances match as a binary classification, and speakers in the testing set may not appear in the training set. Thus, ASV is more challenging than SID.

The meta data for ASV can be downloaded from here.

Training -

fp="<data_root>/kb_data_clean_wav/<lang>" vmd="meta_data/<lang>/valid_data.txt" tknmd="meta_data/$lang/test_known_data.txt" tunkmd="meta_data/$lang/test_data.txt" python s3prl/run_downstream.py \ -n "asv_indicw2v_$lang" \ -m train -u wav2vec2_local \ -d indic_asv -k $pret_model\ -o "config.downstream_expert.datarc.file_path=$fp,, \ config.downstream_expert.datarc.valid_meta_data=$vmd,, \ config.downstream_expert.datarc.test_meta_data=$tknmd,, \ config.downstream_expert.datarc.test_unk_meta_data=$tunkmd,, \ config.downstream_expert.datarc.lang=<lang>"

Evaluate -

l=[] fp="<data_root>/kb_data_clean_wav/<lang>" vmd="meta_data/<lang>/valid_data.txt" tknmd="meta_data/$lang/test_known_data.txt" tunkmd="meta_data/$lang/test_data.txt" override="config.downstream_expert.datarc.eval_batch_size=1,,config.downstream_expert.datarc.file_path=$fp,,config.downstream_expert.datarc.valid_meta_data=$vmd,,config.downstream_expert.datarc.test_meta_data=$tknmd,,config.downstream_expert.datarc.test_unk_meta_data=$tunkmd,,config.downstream_expert.datarc.lang=$lang" bash s3prl/downstream/indic_asv/test_expdir.sh <trained_checkpoint> $l $override

Automatic Speech Recogntion

Automatic Speech Recognition, ASR is the task of transcribing a given audio utterance. The evaluation metric is word error rate (WER).

We use the fairseq library to train ASR models. We provide the data manifest files and the character dictionaries.

• Manifest files - <data_root>/kb_data_clean_wav/<lang>/train/manifest
• Dicts - s3prl/downstream/indic_asr/dicts

For detailed instructions on training ASR models, refer to IndicWav2vec.

Query By Example

Query by Example Spoken Term Detection, QbE detects a spoken term (query) in an audio database (documents) by binary discriminating a given pair of query and document into a match or not. The evaluation metric is maximum term weighted value (MTWV) which balances misses and false alarms.

Download the Query by Example dataset from here[3GB].

Evaluate -

Modify lines 1 and 2 in superb/qbe_new/qbe_indicsuperb/SCORING/score_lang_agg.sh.

bash qbe_indicsuperb/SCORING/score_lang_agg.sh

Citing our work

If you are using any of the resources, please cite the following article:

@misc{https://doi.org/10.48550/arxiv.2208.11761, doi = {10.48550/ARXIV.2208.11761}, url = {https://arxiv.org/abs/2208.11761}, author = {Javed, Tahir and Bhogale, Kaushal Santosh and Raman, Abhigyan and Kunchukuttan, Anoop and Kumar, Pratyush and Khapra, Mitesh M.}, title = {IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages}, publisher = {arXiv}, year = {2022}, copyright = {arXiv.org perpetual, non-exclusive license} }

We would like to hear from you if: - You are using our resources. Please let us know how you are putting these resources to use. - You have any feedback on these resources.

License

Dataset

The IndicSUPERB dataset is released under this licensing scheme: - We do not own any of the raw text used in creating this dataset. The text data comes from the IndicCorp dataset which is a crawl of publicly available websites. - The audio transcriptions of the raw text and labelled annotations of the datasets have been created by us. - We license the actual packaging of all this data under the Creative Commons CC0 license (“no rights reserved”). - To the extent possible under law, AI4Bharat has waived all copyright and related or neighboring rights to the IndicSUPERB dataset. - This work is published from: India.

Code and Models

The IndicSUPERB code and models are released under the MIT License.

Contributors

• Tahir Javed
• Kaushal Bhogale
• Abhigyan Raman
• Anoop Kunchukuttan
• Mitesh Khapra
• Pratush Kumar

Contact

• Anoop Kunchukuttan (anoop.kunchukuttan@gmail.com)
• Mitesh Khapra (miteshk@cse.iitm.ac.in)
• Pratyush Kumar (pratyush@cse.iitm.ac.in)

Acknowledgements

We would like to thank the Ministry of Electronics and Information Technology (MeitY) of the Government of India and the Centre for Development of Advanced Computing (C-DAC), Pune for generously supporting this work and providing us access to multiple GPU nodes on the Param Siddhi Supercomputer. We would like to thank the EkStep Foundation and Nilekani Philanthropies for their generous grant which went into hiring human resources as well as cloud resources needed for this work. We would like to thank DesiCrew for connecting us to native speakers for collecting data. We would like to thank Vivek Seshadri from Karya Inc. for helping setup the data collection infrastructure on the Karya platform. We would like to thank all the members of AI4Bharat team in helping create the Query by Example dataset.