Audio vocal commands recognition demo using Deep Learning. (0-9) digits spotting on Cortex-M Microcontrollers. This project is aiming to provide an example of audio application running on ultra low-power microcontrollers. All the steps, including dataset building, AI model designing, training and integrating on MCU will be detailed. This application could be deployed in any situation where an user has to interact with numbers. Especially in situation where you shouldn't touch any interface and avoid spreading any bacteria. In smart building (lifts, parking), smart home (TV, air conditioner) and in any place where you should keep a good hygiene (hotel, hospital etc).
Python version 3.x.x
Following python libraries are required:
Numpy 1.16.2
Matplotlib 3.0.3
Librosa 0.6.3
Keras 2.2.4
You can use the following instructions to install the python libraries.
TODO
TODO
Clone the GitHub project with the following command line. (Be sure that Git is well installed on your development environment)
git clone https://github.com/nMoussa/audioDigits.git
The spoken digits dataset contains 2000 audio recordings (200 for each digit). Those audio signals was recorded at 8 kHz (mono) under .wav format. The digits are pronounced in English. For the dataset building, 90% of the data is used for the training and 10% for the test (balanced).
The first step is to read the data from the repositories train_digits and test_digits. Then you can select which kind of pre-processing you want to apply and finally save the training and testing data inside a new folder created on purpose.
The main script used for this step is dataset.py.
Start with python dataset.py -h
usage: dataset.py [-h] [-pre {fft,spectrogram,mfcc}] [-ns [NOISE]]
[-sh [SHIFT]] [-pi [PITCH]] [-st [STRETCH]]
optional arguments:
-h, --help show this help message and exit
-pre {fft,spectrogram,mfcc}, --preprocessing {fft,spectrogram,mfcc}
Preprocessing to apply. "fft", "spectrogram", "mfcc"
-ns [NOISE], --noise [NOISE]
data augmentation ratio for noise. Between 0 and 1
-sh [SHIFT], --shift [SHIFT]
data augmentation ratio for shift. Between 0 and 1
-pi [PITCH], --pitch [PITCH]
data augmentation ratio for pitch. Between 0 and 1
-st [STRETCH], --stretch [STRETCH]
data augmentation ratio for stretch. (not enable)
As it mentioned above, there are three kinds of audio pre-processing. The first one is the well known Fast Fourier Transform applied on the audio raw signal which also contains an absolute operation and a normalization. Then, the Log Mel-Spectrogram is computed using some mel filters and scale conversion on the FFT signal. Finally, the DCT (inverse FFT using Discrete Cosine Transform) used on top of the Log Mel-Spectrogram will give the Mel-Frequency Cepstral Coefficients (MFCCs), those features are more representative of the human auditory system and how they catch and recognize a sound.
For more details, on the audio pre-processing, you can have a look on the following links.
Data augmentation is a common approach to increase the baseline dataset and make the AI model more robust. It also solves the overfitting issue in some cases. In this project, we propose four audio data augmentations: noise shift pitch stretch. For each data augmentation, there are two parameters which can be customized.
The first parameter is to decide the proportion of data which will be used to apply the data augmentation. It means that we can select a ratio which will represent the percentage of the initial data to be augmented. For instance, if the noise ratio is set at 0.2, then noise will be added to 20% of the initial data (selected randomly).
The second parameter is to set the level of noise shift pitch stretch added to the initial data. For each augmentation, the factor parameter can be changed. For example, if the noise factor is higher, the signal will be more noisy and then, it will be more challenging for the AI model to recognize the interesting patterns and to classify them.
Adding noise for data augmentation with a noise factor of 0.02.
Adding pitch for data augmentation with a pitch factor of 0.01.
Adding shift for data augmentation with a shift factor of 0.1.
The pre-processing task is based on mathematical operations applied on the raw signal in order to clean and extract relevant features. This is a key step in this project, since, it will impact directly the AI model complexity, memory footprint and the power consumption on the MCU. The aim is finally to find the best tradeoff between model performance and complexity.
Here, all the different pre-processing steps will be showed.
The FFT signal is computed using the Numpy library (https://docs.scipy.org/doc/numpy/reference/generated/numpy.fft.fft.html#numpy.fft.fft). The symmetric part of the FFT signal is removed and the absolute function as well as a normalization are processed on the signal.
The "Mel" scale is a transformation of the frequency scale where the idea is to have a more efficient representation of the signal, considering the way how humans distinguish the sound at different frequencies. To perform this "spectrum scale conversion", some "Mel filters" are defined regarding the input signal specification. "Librosa" library is providing a simple high-level function to perform the Log Mel-Spectrum from a raw audio signal.
The Mel Frequency Cepstral Coefficients (MFCCs) are the key values and peaks of the audio signal. A DCT (discrete cosine function) is applied on the Log Mel-Spectrum to get the MFCCs.
The input data shape will depend on the audio pre-processing used.
| FFT | Spectrogram | MFCC | |
|---|---|---|---|
| Initial input | (2048) | (128, 9) | (13, 9) |
| Reshaped input | (64, 32, 1) | (36, 32, 1) | (13, 9, 1) |
The reshaped dimensions for each pre-processing are just proposed as examples. The user can customize them in the audio_constants.py. However, the reshaped input must match with the initial input. The 2D shape has been considered because of the use of 2D convolutional filters during the model designing. The input size is a key point which directly impacts the AI model global complexity. Thus, the same AI model trained with FFT signal will be much more complex than the model trained with MFCC.
With the following command line, an user can create four Numpy files (x_train.npy, y_train.npy, x_test.npy and y_test.npy ) representing the train and test sets.
python dataset.py
Executing this script will create a new folder in your current repository and it will save the Numpy files inside the created folder. User is free to adjust the following arguments summarized as follow.
| Arguments | Choices | By default | Recommended |
|---|---|---|---|
| -pre --preprocessing | fft, spectrum, mfcc |
fft |
mfcc |
| -ns --noise | between 0-1 (float) |
0 |
0.2 |
| -pi --pitch | between 0-1 (float) |
0 |
0.2 |
| -st --shift | between 0-1 (float) |
0 |
0.2 |
Command example with the recommended arguments' values.
python dataset -pre mfcc -ns 0.2 -pi 0.2 -st 0.2
Each time, the dataset.py script is called, a new folder is created and the Numpy files are stored inside this folder.
As example:
$ python dataset.py -pre mfcc
INFO:root: _Main entry point_
INFO:root: Data augmentation with noise by 0%
INFO:root: Data augmentation with shift by 0%
INFO:root: Data augmentation with pitch by 0%
INFO:root: Training set --> input shape (1800, 13, 9, 1) and output shape (1800,)
INFO:root: Data augmentation with noise by 0%
INFO:root: Data augmentation with shift by 0%
INFO:root: Data augmentation with pitch by 0%
INFO:root: Test set --> input shape (200, 13, 9, 1) and output shape (200,)
INFO:root: Data saved in new folder "xxx/audioDigits/mfcc_log_fold_47_1358"
$ ls mfcc_log_fold_47_1358/
x_test.npy x_train.npy y_test.npy y_train.npy
This new folder will be used with the model.py script in the next section.
In this section, the deep learning subject will not be fully detailed. I recommend you to follow some online courses to learn more about this approach.
A deep learning model can be seen as a black box which will consider some input data (audio data), it will then apply a mathematical processing to compute some output data (digit number). This process is called "AI model inferencing" and this is our application end target. Before, reaching this goal, the deep learning model needs to be designed and trained, this is what we will see in the next parts.
Artificial neural networks are powerful algorithms which can learn some experiences from existing data, then, they can improve those experiences and reuse them with new data. In this part, two popular neural network topologies will be considered: "Fully Connected Neural Network" (also called Dense or Multiple Perceptron) and "Convolutional Neural Network". Obviously both topologies should be designed in a such way that they would fit into small tiny devices such as microcontrollers. Thus, for each topology, the memory footprint will be analyzed as well as the complexity in term of operations.
As the audio input data format will impact the model in a global way (performance & complexity), all the different audio pre-processing will be tested with the two following topologies.
The proposed Dense neural network is composed by five dense layers where the first four layers have a size of sixteen neurons and the last dense layer has a size of ten neurons because there are ten digits to classify. This architecture is popular for classification tasks.
The proposed CNN is composed by four convolutional layers and two dense layers. Two pooling layers are used to reduce the tensors' sizes and increase the invariance. The first stage with convolutional layers is performing the features extraction and then, the second stage of dense layers is classifying the extracted features. This architecture is popular where you want to find, extract, learn interesting features (from the input data) and classify them.
In both topologies, a dropout regularization layer is also used during the training to avoid overfitting. The two model can be customized in the following script model.py.
In general, the model training required a lot of resources in term of computation standpoint. Cloud platforms and GPUs are mainly used to perform this task. For our application, a simple CPU will be enough.
There are several AI frameworks which make the AI model designing and training easy and friendly. But, you still need to understand the basic concepts such as training parameters, model hyperparameters and how to fine-tune an AI model. Moreover, the AI models will be integrated on microcontrollers, so, they need to be optimized for such devices. Thus, our project required an AI framework which would be supported or would provide the support for microcontrollers. This is why Keras and TensorFlow Lite are selected as AI framework to design and train the AI models.
Keras is one of the most simple framework to quickly design and train a model. TensorFlow Lite is providing a converter tool to generate optimized code for microcontrollers and it can support a model from Keras. Later, other AI frameworks/tools would be considered to evaluate and find the best model integration on microcontrollers for our use case.
The python script model.py is provided to design and train a model in a row.
$ python model.py -h
Using TensorFlow backend.
usage: model.py [-h] [-m {CNN,DENSE}] [-df [DATA_FOLDER]]
optional arguments:
-h, --help show this help message and exit
-m {CNN,DENSE}, --model {CNN,DENSE}
Neural network model to use. "CNN" or "DENSE"
-df [DATA_FOLDER], --data_folder [DATA_FOLDER]
Data folder where the training and test sets are
stored
A first parameter --model allows to select which topology you want to select (CNN or Dense).
A second parameter --data_folder is mandatory, since, it's indicating the folder where the Numpy data files are stored.
With the following command example, it will design a Dense model and train it on the MFCC audio data inside the indicated folder.
python model.py -m DENSE -df mfcc_log_fold_47_1358
The training parameters as the number of epochs, the batches size and the learning rate are defined inside the script data_constants.py. User can customize them. By default, the epoch is fixed at 20, the batch size at 32 and the learning rate at 0.5e-3. In order to be aligned for the evaluation purpose, all those parameters don't change for the training. But, it highly recommended to customize those parameters to improve the model performance.
This table below is wrapping-up the models training on different audio input format. The accuracy criteria is the model performance on the test set which represents 10% of the overall dataset (randomly chosen). The memory footprint is based on the number of training parameters (weights/biases) and the activation buffers. The complexity represents the number of multiplications and additions required for one inference. No data augmentation is used for this part, a dropout layer is used just after the feature extraction done by the convolutional layers to avoid overfitting.
| FFT - CNN | FFT - Dense | Spec - CNN | Spec - Dense | MFCC - CNN | MFCC - Dense | |
|---|---|---|---|---|---|---|
| Accuracy (%) | ~ 92 | ~ 88 | ~ 94 | ~ 44 | ~ 96 | ~ 98 |
| Memory footprint (Flash/RAM) | 144/198 (Kbytes) | 256/264 (Kbytes) | 80/113 (Kbytes) | 48/148 (Kbytes) | 20/11 (Kbytes) | 76/15 (Kbytes) |
| Complexity (MACC) | 5.35 M | 0.96 M | 2.92 M | 0.54 M | 0.29 M | 0.12 M |
The idea is to analyze the tendencies because the exact values depend on how you train your model.
A first point about the Dense model is that it doesn't provide strong performance with FFT signal and spectrogram. This can be explained by the fact that from both audio pre-processing, interesting features still need to be extracted and Dense topology is not suitable for this task. Therefore, the Dense is highly well performing for the MFCCs. Those are already extracted features which may only require to be classified. The second point about the CNN model is that the accuracy is improving when the audio-preprocessing is becoming more complex. In addition, the complexity and memory footprint are decreasing at the same time.
In the overall, thanks to the audio pre-processing, the memory footprint and the complexity of the models can be optimized. However, the pre-processing also required computation bandwidth., this is why an investigation on the pre-processing should be done in order to figure out if it's really worth to use it. The models trained with the MFCCs is giving not only high accuracies, but it's also dramatically reducing the global model complexity.
Before going too deep in the C project, a tool is proposed by STMicroelectronics to quickly analyze an AI model on microcontrollers STM32. The tool is called X-Cube-AI and it's provided with the STM32CubeMX environment.