Evaluate_generator produces wrong accuracy scores?

[skoch9]

Hello, I run a slightly modified version of the keras fine tuning examples which only fine tunes the top layers (with Keras 2.0.3/Tensorflow on Ubuntu with GPU). This looks like the following:

img_width, img_height = 150, 150
train_data_dir = 'data/train_s'
validation_data_dir = 'data/val_s'
nb_train_samples = 2000
nb_validation_samples = 800
epochs = 10
batch_size = 16

base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_width, img_height, 3))

top_model = Sequential()
top_model.add(Flatten(input_shape=base_model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dense(1, activation='sigmoid'))

model = Model(inputs=base_model.input, outputs=top_model(base_model.output))
model.compile(loss='binary_crossentropy', optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])

train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode='binary', shuffle=False)

model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size,
    verbose=2, workers=12)

score = model.evaluate_generator(validation_generator, nb_validation_samples/batch_size, workers=12)

scores = model.predict_generator(validation_generator, nb_validation_samples/batch_size, workers=12)

correct = 0
for i, n in enumerate(validation_generator.filenames):
    if n.startswith("cats") and scores[i][0] <= 0.5:
        correct += 1
    if n.startswith("dogs") and scores[i][0] > 0.5:
        correct += 1

print("Correct:", correct, " Total: ", len(validation_generator.filenames))
print("Loss: ", score[0], "Accuracy: ", score[1])

With this, I get unreliable validation accuracy results. For example, predict_generator predicts 640 out of 800 (80%) classes correctly whereas evaluate_generator produces an accuracy score of 95%. Someone in #3477 suggests to remove the rescale=1. / 255 parameter from the validation generator, then I get results of 365/800=45% and 89% from evaluate_generator.

Is there something wrong with my evaluation or is this due to a bug? There are many similar issues (e.g. #3849, #6245) where the stated accuracy (during training and afterwards) doesn't match the actual predictions. Could someone experienced maybe shine some light onto this problem? Thanks

[joeyearsley]

Having looked into the backend I believe this is due to the number of workers if you have more than one worker there is nothing to ensure consistency of file loading across them. Therefore it is possible that multiple files are shown numerous times in these methods, as the 12 generators are randomly initialised but don't actually share a file list state.

Maybe @fchollet or @farizrahman4u can confirm?

[skoch9]

Thanks for your answer. I already considered that the image generators are not threadsafe. However, I was able to reproduce the problem with setting all three worker counts to 1. Does the order of the filenames maybe not correspond to the order of the scores?

Recently added similar issues are #6540 and #6544.

[skoch9]

Plus, @fchollet mentioned here that the ImageDataGenerator supports multiprocessing.

[avn3r]

@skoch9 try to set pickle_safe=True. As @joeyearsley mention, for me it had to do with worker > 1. I am actually running on a custom version of Keras where I make evaluate_generator inside fit_generator workers=1. That way I can train with multiple workers but predict/evaluate with a single worker.

@fchollet Please make evaluate_generator, and predict_generator workers=1 always, or eliminate parameter until it is fixed.

Make sure:

1. shuffle = false
2. pickle_safe = True
3. workers = 1

Let me know if that gives you consistent results.

[skoch9]

Thanks @abnerA for your answer. I investigated a little further and found that running evaluate_generator before predict_generator without setting pickle_safe=True messes up the predictions of the latter, even without multiprocessing.

score = model.evaluate_generator(validation_generator, nb_validation_samples/batch_size, pickle_safe=False)
scores = model.predict_generator(validation_generator, nb_validation_samples/batch_size)

As for the parameters, when setting workers >1, shouldn't pickle_safe automatically be set to True also for the fit_generator?

[avn3r]

@skoch9 no. Pickle safe is false by default in all keras generator methods. The only difference between this two modes is False uses multithread and True uses multiprocess.

[skoch9]

Sure, but it doesn't make sense to allow invalid parameter configurations. And I would consider it also problematic (->a bug) that running evaluate_generator before predict_generator changes the prediction results.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[romainVala]

Hello
I made a test based on the mnist_cnn.py example.
I just change model.fit by

gen=matrix_generator(x_train,y_train,batch_size=batch_size) val_steps = len(y_test)/batch_size

model.fit_generator(gen,epochs=epochs, steps_per_epoch= steps_per_epoch, validation_data=(x_test, y_test))

with the generator define to take subpart of the input matrix

def matrix_generator(x,y,batch_size=1,validation=False):
import numpy as np
item=0;
tot_item,dimx,dimy,dimz=x.shape

X = np.zeros((batch_size,dimx,dimy,dimz), dtype=np.float32)

if len(y.shape)==1:
    Y = np.zeros((batch_size))
else:
    Y = np.zeros((batch_size, y.shape[1]))

while True:
    for bs in range(batch_size):
        ximg = x[item,:,:,:]
        
        yout = y[item,:]                
        yout = yout.reshape(1,yout.shape[0])
        Y[bs,:] = yout
        
        X[bs,:,:,:] = ximg

        item +=1
        if item>tot_item-1:
            item=0

    if validation:
        yield X
    else:
        yield X,Y

`
Now if I test the model

model.evaluate(x_test, y_test, verbose=0)
Out[18]: [0.16519621040821075, 0.96389999999999998]

but
gen_test = matrix_generator(x_test,y_test,batch_size=batch_size)
model.evaluate_generator(gen_test,len(y_test)/batch_size,workers=1)
Out[17]: [0.31920816290837067, 0.93760016025641024]

same learn parameter, same test data, but different results

What to do ???
Many thanks for your help

[romainVala]

To follow up
I get the correct result only if I set
max_q_size=1

if I only set worker=1 it does not woks either (and give different results each time)

Hi,
I am having a similar problem with a binary classifier that uses 2 outputs. model.evaluate_generator() and model.predict() both suggest that I have 50% accuracy (chance) and a loss of over 1.0, while model.fit_generator() always gives me a loss of under 0.5 and accuracy of over 80%.

for x in range(10000):
    print("starting epoch {0}".format(x))
    mg_acc = mg_model.evaluate_generator(mg_gen, 2, 
        max_queue_size=1, workers=1, use_multiprocessing=False)
    mg_model.fit_generator(mg_gen, 2, epochs=1, callbacks=mg_callbacks, 
        max_queue_size=1, workers=1, use_multiprocessing=False, verbose=1)
    print(mg_acc)
    print(mg_model.metrics_names)

It does not matter if use_multiprocessing is True or False, or the order of the calls (fit_generator before evaluate_generator or vice-versa), or if one call is commented out.

I know that this code will not result in identical generator output (each calls the next iteration of the generator) but the output of the generator is similar between iterations, and the evaluate_generator is always the one that does poorly, regardless of order.

EDIT: SOLVED: My accuracy difference was eliminated when I removed all batch normalization layers (also removed dropout layers, but I don't think that was the cause)

[GalAvineri]

I too have a big difference between the reported fit_generator results and the later evaluate_generator results.

I've looked into this a bit and found the following results:

1. When i use evaluate_generator with a generator that does not shuffle the suite, i get results that are
   very different than those reported by fig_generator.
   But when i use evaluate_generator with a generator that does shuffle the suite, i get results that are
   similiar to those reported by fit_generator.

2. When i use evaluate (without any generators) the output is exactly the same as evaluate_generator
   without shuffling

3. When i use model.predict and infer the measurements manually, i get the same measurements
   reported by fit_generator (and the same results as evaluate_generator with shuffling)

Can anyone verify that any of the above happen to them as well?

A note:
I use a very simple model - just one dense layer.
no dropouts or batch normalizations to create any doubts as mentioned by @jeremydr2