Real-time decoding

[ericbolo]

What would be the main steps for building a real-time decoder on top of EESEN?

I read in the EESEN paper that composing the tokens, lexicon and grammar speeds up decoding a great deal, and I'd like to leverage that in a real-time context: capture an audio stream and output the transcripts progressively.

Is that by any chance in the works?

If not, I could try and give it a shot.

Thank you for this great project

[fmetze]

Eric,

thanks for the flowers. The main problem is the use of the bi-directional LSTM as an acoustic model, which in theory requires you to have the while segment available before evaluating the acoustic model, i.e. no run-on or streaming capability like you want.

There are several papers on how to get around this, because all of the really important information for ASR seems to be contained in a 200ms window, so not all the right context is needed. Within our Tensorflow implementation, we want to try a CNN for that, rather than the BLSTM, so we could get around that limitation easily, but we have not gotten round to implement that. The second challenge is of course to implement the code for streaming the audio and actually processing it. There is probably some Kaldi code that we could leverage.

If you're interested, this would be a great addition to Eesen. We're using the fast decoding and small memory footprint for processing large amounts of data, but you're right, it would go well with real-time recognition, too. Want to try?

Let me know what you think!

[ericbolo]

Thank you for your response!

With the current BiLSTM setup, what would be required is then a feature window that slides over the input and includes for each input all the left context and just enough of the right context. Is that correct?

Kaldi does have boilerplates and tools for online decoding: http://kaldi-asr.org/doc/online_decoding.html

In that web page they also mention per-speaker cepstral mean and variance normalization (CMVN), which I see is being used in the example EESEN scripts. What I have in mind is a system that takes any user's voice and outputs a transcript, without necessarily knowing who the speaker is or having access to past utterances. Any challenges to foresee?

[ericbolo]

For online decoding with neural nets, Kaldi recommends constructing an i-vector that summarizes speaker properties, and training the neural net with audio features + i-vector. In the absence of past utterances, the i-vector is built from the audio from time 0 to some time t. So there would be an additional delay from building the i-vector, if I understand correctly.

From http://kaldi-asr.org/doc/online_decoding.html
"
Our best online-decoding setup, which we recommend should be used, is the neural net based setup. The adaptation philosphy is to give the neural net un-adapted and non-mean-normalized features (MFCCs, in our example recipes), and also to give it an iVector. ... Our idea is that the iVector gives the neural net as much as it needs to know about the speaker properties. This has proved quite useful. The iVector is estimated in a left-to-right way, meaning that at a certain time t, it sees input from time zero to t. It also sees information from previous utterances of the current speaker, if available.
"

[fmetze]

Yes, that is by and large correct.

The big challenge is speaker diarization, unless you only have one speaker in your audio channel. Imagine you have two speakers, a loud male and a soft female, in the same channel of the recording that you want to recognize. Even if they don't overlap, you need to separate them somehow, either by estimating and updating the corresponding cepstral means, or the corresponding i-vectors.

I could imagine that a neural network learns the normalization properties from the i-vectors, but I am wondering if that doesn't introduce all other types of failure modes like means not being 0, etc.

In sum, the BLSTM problem is just one problem that you need to solve for on-line recognition. Can you see a solution that would work for you for the above issues, so we can think about the BLSTM issue specifically?

[ericbolo]

Ok, I now have a pretty good understanding of the diarization/speaker normalization issues, none of them insurmountable in my application.

For now, I can focus on the decoding of the BLSTM outputs. You mentioned ASR probably only needs a 200ms window, any papers you could point me to?

[fmetze]

http://www.asru2015.org/Papers/ViewPapers.asp?PaperNum=1103 <http://www.asru2015.org/Papers/ViewPapers.asp?PaperNum=1103>

On Aug 4, 2017, at 5:37 PM, ericbolo ***@***.***> wrote: Ok, I now have a pretty good understanding of the diarization/speaker normalization issues, none of them insurmountable in my application. For now, I can focus on the decoding of the BLSTM outputs. You mentioned ASR probably only needs a 200ms window, any papers you could point me to? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEnA8WwtkAT11TPQIYGznLe62D0uj3esks5sUzq1gaJpZM4OnXsl>.

Florian Metze <http://www.cs.cmu.edu/directory/florian-metze> Associate Research Professor Carnegie Mellon University

[ericbolo]

Thank you for the link.

I gave it a quick read, my only worry is that in the paper the data is pre-segmented with GMM/HMM, and the training does not use CTC loss.

Could the 0.5 s spectral window be too small for CTC loss training?

[ericbolo]

Elaborating: from the paper I understand they sub-sample the data to avoid overfitting, so we don't have access to all the outputs of the utterance, possibly hampering CTC loss.

[fmetze]

Right, the paper does not use CTC loss, but I don't think this would matter much, certainly not for the LSTMs, which is where we have the recurrent connections. CTC affects the way the error signal is computed during training, but during inference, the computation is straightfoward and independent of the window. Except of course that we do need a segmentation - which is the other problem that you mention. The exact value of the window, 2*0.2 or 0.5 seconds would have to be determined through experiments, of course.

[ericbolo]

I've stumbled on this paper, which proposes a kind of BLSTM that is compatible with online decoding: http://ieeexplore.ieee.org/document/7953176/

Thought it might be of interest

[fmetze]

Yes, it is of interest - are you still trying to look into this and maybe implement this or something similar? I was hoping to do something here during the semester,er but it seems we don’t have enough hands as is ...

…

On Jan 17, 2018, at 8:10 AM, ericbolo ***@***.***> wrote: I've stumbled on this paper, which proposes a kind of BLSTM that is compatible with online decoding: http://ieeexplore.ieee.org/document/7953176/ <http://ieeexplore.ieee.org/document/7953176/> Thought it might be of interest — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEnA8V0GkW_0ykppcY4wCW2XXjmGPcHVks5tLfEogaJpZM4OnXsl>.

[ericbolo]

I'm still very much interested in online decoding, yes, but unfortunately my hands are full this month and the next. If anyone is interested to work on this with me starting early April, do let me know !

…

On 7 February 2018 at 04:48, Florian Metze ***@***.***> wrote: Yes, it is of interest - are you still trying to look into this and maybe implement this or something similar? I was hoping to do something here during the semester,er but it seems we don’t have enough hands as is ... > On Jan 17, 2018, at 8:10 AM, ericbolo ***@***.***> wrote: > > I've stumbled on this paper, which proposes a kind of BLSTM that is compatible with online decoding: http://ieeexplore.ieee.org/ document/7953176/ <http://ieeexplore.ieee.org/document/7953176/> > Thought it might be of interest > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub < #141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEnA8V0GkW_ 0ykppcY4wCW2XXjmGPcHVks5tLfEogaJpZM4OnXsl>. > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEXQ_GeDkfDIdTRJY9gZT2-kkb68Rd71ks5tSRz7gaJpZM4OnXsl> .

-- Eric Bolo CTO tel: 06 29 72 19 80 <http://06%2058%2079%2025%2037/>

______________________ Batvoice Technologies 10, rue Coquillière - 75001 Paris www.batvoice.com

[fmetze]

yep, help wanted! probably even in April!

…

On Feb 7, 2018, at 3:06 AM, ericbolo ***@***.***> wrote: I'm still very much interested in online decoding, yes, but unfortunately my hands are full this month and the next. If anyone is interested to work on this with me starting early April, do let me know ! On 7 February 2018 at 04:48, Florian Metze ***@***.***> wrote: > Yes, it is of interest - are you still trying to look into this and maybe > implement this or something similar? I was hoping to do something here > during the semester,er but it seems we don’t have enough hands as is ... > > > On Jan 17, 2018, at 8:10 AM, ericbolo ***@***.***> wrote: > > > > I've stumbled on this paper, which proposes a kind of BLSTM that is > compatible with online decoding: http://ieeexplore.ieee.org/ > document/7953176/ <http://ieeexplore.ieee.org/document/7953176/> > > Thought it might be of interest > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub < > #141 (comment)>, or mute > the thread <https://github.com/notifications/unsubscribe-auth/AEnA8V0GkW_ > 0ykppcY4wCW2XXjmGPcHVks5tLfEogaJpZM4OnXsl>. > > > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#141 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AEXQ_GeDkfDIdTRJY9gZT2-kkb68Rd71ks5tSRz7gaJpZM4OnXsl> > . > -- Eric Bolo CTO tel: 06 29 72 19 80 <http://06%2058%2079%2025%2037/> ______________________ Batvoice Technologies 10, rue Coquillière - 75001 Paris www.batvoice.com — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEnA8RoDIc7MNRQYvvs2rN68-g9yPWWbks5tSVmWgaJpZM4OnXsl>.

[efosler]

What's the current status on this? I'm starting the (crazy) sabbatical project and it's pretty clear that some sort of online decoding mechanism is going to be necessary. I can probably pitch in to help but I'll be shaking the rust off of my coding skills. @ericbolo, @fmetze any interest in this?