ML Examples with tf.data

[niemiaszek]

I'm looking for a library to use with my next data augmentation pipeline. My target use is for speech oriented task, so I would like to augment my clean speech audio (only speakers voice with no background noise) with mostly background noises, RIRs, phase shift and lossy compression.

For now I considered offline dataset generation, where in multiprocessing setup I first load wavs, augment them and then proceed with usual preprocessing (wav -> mel-spectrogram with tf.io) to prepare tfrecords further used in training pipeline with tf.data API.

I was looking audiomentations which looks fine for my use case, but I stumbled upon pedalboard and got really interested, given how it is oriented on performance and high-quality effects. It's stated that this library is used by Spotify internally for ML training with TF, but I can't find any examples here on this use case.

If processing was fast enough one could easily setup online tf.data training pipeline with augmentations. My question is if you would consider adding some examples of how to setup such data pipeline?

[psobot]

Hi @niemiaszek! While we don't have an example of using Pedalboard with tf.data in our README, Pedalboard does include example code in its tests:

pedalboard/tests/test_tensorflow.py

Lines 34 to 53 in f2c2ccd

	def test_can_be_called_in_tensorflow_data_pipeline():
	import tensorflow as tf
	sr = 48000
	plugins = pedalboard.Pedalboard([pedalboard.Gain(), pedalboard.Reverb()])
	noise = np.random.rand(sr)
	ds = tf.data.Dataset.from_tensor_slices([noise]).map(
	lambda audio: tf.numpy_function(plugins.process, [audio, sr], tf.float32)
	)
	model = tf.keras.models.Sequential(
	[tf.keras.layers.InputLayer(input_shape=(sr,)), tf.keras.layers.Dense(1)]
	)
	# Train a dummy model just to ensure nothing explodes:
	model.compile(loss="mse")
	model.fit(ds.map(lambda effected: (effected, 1)).batch(1), epochs=10)
	assert np.allclose(plugins(noise, sr), np.array([tensor.numpy() for tensor in ds]))

I'll leave this issue open until we do add a similar block of code to our documentation, however, as asking users to read the tests is not the most discoverable. 😅