Please fill out the form below.
System Information
- Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans):Tensorflow / Keras
- Framework Version:1.11
- Python Version:2
- CPU or GPU:GPU
- Python SDK Version:
- Are you using a custom image:No
Describe the problem
I am getting an error that the global steps are not increasing. Further, the training seeps to have a Warm Start (dont know why, or if thats a problem at all)
Minimal repro / logs
2018-10-26 07:24:04,680 INFO - tensorflow - loss = 0.5054888, step = 0 2018-10-26 07:24:12,735 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize. 2018-10-26 07:24:18,908 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize. 2018-10-26 07:24:25,089 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize. 2018-10-26 07:24:31,263 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize. 2018-10-26 07:24:39,229 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize. 2018-10-26 07:24:39,230 INFO - tensorflow - loss = 0.5811208, step = 0 (34.550 sec) 2018-10-26 07:24:45,407 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.
This is happening on any keras_model_fn, even then one in examples.
Please fill out the form below.
System Information
Describe the problem
I am getting an error that the global steps are not increasing. Further, the training seeps to have a Warm Start (dont know why, or if thats a problem at all)
Minimal repro / logs
2018-10-26 07:24:04,680 INFO - tensorflow - loss = 0.5054888, step = 0 2018-10-26 07:24:12,735 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize. 2018-10-26 07:24:18,908 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize. 2018-10-26 07:24:25,089 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize. 2018-10-26 07:24:31,263 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize. 2018-10-26 07:24:39,229 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize. 2018-10-26 07:24:39,230 INFO - tensorflow - loss = 0.5811208, step = 0 (34.550 sec) 2018-10-26 07:24:45,407 WARNING - tensorflow - It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 0 vs previous value: 0. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.This is happening on any keras_model_fn, even then one in examples.