:zap: FINETUNE WAV2VEC 2.0 FOR PINYIN STYLE SPEECH RECOGNITION

Table of contents

1. Documentation
2. Available Features
3. Installation
4. Train
5. Inference
6. Logs and Visualization

Documentation

Suppose you need a simple way to fine-tune the Wav2vec 2.0 model for the task of Speech Recognition on your datasets, then you came to the right place.
All documents related to this repo can be found here: - Wav2vec2ForCTC - Tutorial

Available Features

• [x] Multi-GPU training
• [x] Automatic Mix Precision
• [ ] Push to Huggingface Hub (Offline)

Installation

pip install -r requirements.txt

Train

1. Prepare your dataset
   • Prepare the folder data for both wav files and txt files pair with same name
   • path and transcript columns are compulsory. The path column contains the paths to your stored audio files, depending on your dataset location, it can be either absolute paths or relative paths. The transcript column contains the corresponding transcripts to the audio paths.
   • Check out our example folders for more information.
   • Important: Ignoring these following notes is still OK but can hurt the performance.
     • Special characters such as r'[,?.!\-;:"“%\'�]' are removed by default, but you can change them in the base_dataset.py if your transcript is not clean enough.
     • If your transcript contains special tokens like bos_token, eos_token, unk_token (eg: <unk>, [unk],...) or pad_token (eg: <pad>, [pad],...)). Please specify it in the config.toml otherwise the Tokenizer can't recognize them.
2. Configure the config.toml file: Pay attention to the pretrained_path argument, it loads "facebook/wav2vec2-base" pre-trained model from Facebook by default. Change it to the pretrained models from phase 1 if need
3. Run
   • Prepare the datasets : python create_data.py -c config.toml
   • Start training from scratch: python train.py -c config.toml
   • Resume: python train.py -c config.toml -r
   • Load specific model and start training: python train.py -c config.toml -p path/to/your/model.tar

Inference

We provide an inference script that can transcribe a given audio file or even a list of audio files. Please take a look at the arguments below, especially the -f TEST_FILEPATH and the -s HUGGINGFACE_FOLDER arguments: ```cmd usage: inference.py [-h] -f TESTFILEPATH [-s HUGGINGFACEFOLDER] [-m MODELPATH] [-d DEVICEID]

ASR INFERENCE ARGS

optional arguments: -h, --help show this help message and exit -f TESTFILEPATH, --testfilepath TESTFILEPATH It can be either the path to your audio file (.wav, .mp3) or a text file (.txt) containing a list of audio file paths. -s HUGGINGFACEFOLDER, --huggingfacefolder HUGGINGFACEFOLDER The folder where you stored the huggingface files. Check the argument of [huggingface.args] in config.toml. Default value: "huggingface-hub". -m MODELPATH, --modelpath MODELPATH Path to the model (.tar file) in saved/<projectname>/checkpoints. If not provided, default uses the pytorchmodel.bin in the <HUGGINGFACEFOLDER> -d DEVICEID, --deviceid DEVICE_ID The device you want to test your model on if CUDA is available. Otherwise, CPU is used. Default value: 0 ```

Transcribe an audio file: ```cmd python inference.py \ -f path/to/your/audio/file.wav(.mp3) \ -s huggingface-hub

output example:

transcript: Hello World ```

Transcribe a list of audio files. Check the input file test.txt and the output file transcript_test.txt (which will be stored in the same folder as the input file): cmd python inference.py \ -f path/to/your/test.txt \ -s huggingface-hub

Logs and Visualization

The logs during the training will be stored, and you can visualize it using TensorBoard by running this command: ```

specify the in config.json

tensorboard --logdir ~/saved/

specify a port 8080

tensorboard --logdir ~/saved/ --port 8080 ```