Training audioset strong with added custom label

[acul3]

Hey @fschmid56 , thank you for your great work

I am trying to reproducing your training with audioset strong but with added custom label, basically adding new sound file and new label like overlapped speech, multi speaker, new music etc to audioset strong dataset along with time tagging also.

Do you think its possible? Have any suggestion?

[fschmid56]

Hi @acul3!

Thanks for your interest.

What is your goal? Do you really need these classes added to the AudioSet Strong training, or would it be sufficient to fine-tune the models on your specific classes after the AudioSet Strong training?

[acul3]

@fschmid56 thank you for your reply!

i think the first one , basically i want to add these classes on audioset strong., so total label maybe lik 456(audioset strong label) + x (where x is my label),, but since audioset pre training required ensemble pseudo labels (CMIIW) , i doubt to continue this .

[fschmid56]

Conceptually, you could train without the pseudo labels or with pseudo labels, but only for the AS Strong classes and not for your custom classes.

Both strategies are technically feasible, but they require modifying some parts of the dataset, the label encoding, and the training loop.

Can you tell me more about your target application? Then, I might be able to come up with a simpler solution.

[acul3]

@fschmid56 , thank you again

my aplication is to using this model to detect if given chunk audio contain overlapped speech, low speech, or maybe multispeaker, with offset "timestamp" is optional .

but i also it still want it to be able detect classes that already in audioset strong such as music,background noise

maybe i just fine tune the model ?

[fschmid56]

I think the easiest solution you could try is to leave the pre-trained model untouched, so you will be able to detect AudioSet strong classes. For your own classes, you add a separate classification head.

In this file:

PretrainedSED/models/prediction_wrapper.py

Line 203 in d451e38

elif self.head_type == "linear":

Instead of having only a 'strong_head' and 'weak_head', you could have another linear layer called 'custom_head' that outputs predictions for your added classes.

This way, training should be very fast, as you only need to update the weights of a single linear layer.

[acul3]

@fschmid56 appreciate for your help sofar!

is there a script training/finetuning available for this case, to start with ?

I think the easiest solution you could try is to leave the pre-trained model untouched, so you will be able to detect AudioSet strong classes. For your own classes, you add a separate classification head.

In this file:

PretrainedSED/models/prediction_wrapper.py

Line 203 in d451e38

elif self.head_type == "linear":
Instead of having only a 'strong_head' and 'weak_head', you could have another linear layer called 'custom_head' that outputs predictions for your added classes.

This way, training should be very fast, as you only need to update the weights of a single linear layer.