Question about the speed

[HONGJINLYU]

Hi:

Rather nice work!

There is a problem when I run this code: the training process is expensive. One iteration almost needs 11 seconds.

And another weird thing is that the code can still run smoothly, although the following information shows:

Processes are started. Start training. [*] Reading latest checkpoint from folder ./models/1006_water_portrait_try1/checkpoint. [*] Reading latest checkpoint from folder ./models/1006_water_portrait_try1/checkpoint_long. [!] Load failed...

Could you share your time-consumption about the training process or one iteration?

And does this above ''Load failed'' matter?

Best Regards

[HalbertCH]

Thanks for your interest! For the first problem, our model consumes about 57 hours for 300000 iterations (i.e., about 0.684s per iteration). I guess the reason why your training process is so slow is that you are in the CPU mode. Please remember to set your available GPU ID in Line185 of the file ‘main.py’. We are sorry that we didn’t make it clear. We will update our ‘readme.md’ file recently. For the second problem, the ‘Load failed’ matter arises because there is an ‘if-else’ sentence in Line347-353 of the file ‘model.py’. Its effect is that if there is no pre-trained checkpoint files, the model will be trained from scratch. Otherwise, the model will first load the pre-trained weights, and then continue the training process, which is useful when your training process terminates accidentally and you don’t want to retrain your model from scratch (because it is time-consuming).

[prometx11]

@HalbertCH What gpu have you used for training the model? I want to estimate the time and costs it will take to train the model.
Also is there any possibility to skip some time of the training, because if I want to tweak the content/style weights and it takes 50+ hours to train again it will be really hard to tweak the model.

[HalbertCH]

@prometx11 Sorry for the late reply! Our model is trained and tested on NVIDIA GeForce RTX 2080. As far as I know, there is no operation that can be skipped during training. But I think you can try to reduce the number of iterations when tweaking the model.