Stuck on evaluate

[agerewines]

System information

• OS version/distro: Windows 10
• .NET Version (eg., dotnet --info): ML.NET 1.1.0

Issue

• What did you do?
  Created ML model with Model Builder, given 1000 seconds, it was able to evaluate 2 models and returned the best one.
  Afterwards I tried to rerun the modelbuilder class to recreate the model.
  The csv from where I take the data is around 8 MB, with 70k rows
• What happened?
  It has been stuck on Evaluate for so long, 48 mins. EDIT: now 82 mins.
• What did you expect?
  Have some feedback on the evaluation process or create the new model.

My suggestion is to have some logs on the screen while it is evaluating.

Source code / logs

[justinormont]

Related issue: dotnet/machinelearning-modelbuilder#126

Have some feedback on the evaluation process or create the new model.
My suggestion is to have some logs on the screen while it is evaluating.

Your suggestion sounds good to me. Though ML.NET currently has the option of a "firehose" trace log, or nothing at all.

To quote an earlier issue comment:

As mentioned in #3235, MLContext.Log() doesn't have a verbosity selection, so it's more of a firehose.

If a verbosity argument is added to MLContext.Log(), the log output from there should be human readable to see general progress.

I believe it's still hidden within the firehose of output and once the verbosity is scaled down, you should see messages like:

LightGBM objective=multiclassova
[7] 'Loading data for LightGBM' finished in 00:00:15.6600468.
[8] 'Training with LightGBM' started.
..................................................(00:30.58)	0/200 iterations
..................................................(01:00.9)	1/200 iterations
..................................................(01:31.2)	2/200 iterations
..................................................(02:01.4)	2/200 iterations
..................................................(02:31.9)	3/200 iterations
..................................................(03:02.5)	4/200 iterations
..................................................(03:32.9)	4/200 iterations
..................................................(04:03.6)	5/200 iterations
..................................................(04:34.4)	5/200 iterations
..................................................(05:04.8)	6/200 iterations

And naively extrapolating, there's around 2.7 hours left in the LightGBM training.