add automl experiment api && cfo tuner - #6140
Conversation
|
/azp run In reply to: 1087834657 In reply to: 1087834657 |
|
Azure Pipelines successfully started running 2 pipeline(s). In reply to: 1087834863 In reply to: 1087834863 |
LittleLittleCloud
added
the
AutoML.NET
Codecov Report
@@ Coverage Diff @@
## main #6140 +/- ##
==========================================
- Coverage 68.33% 68.29% -0.04%
==========================================
Files 1097 1098 +1
Lines 241915 242080 +165
Branches 25157 25166 +9
==========================================
+ Hits 165317 165333 +16
- Misses 70049 70177 +128
- Partials 6549 6570 +21
Flags with carried forward coverage won't be shown. Click here to find out more.
|
| TrialResult Run(TrialSettings settings); | ||
| } | ||
|
|
||
| internal class BinaryClassificationCVRunner : ITrialRunner |
There was a problem hiding this comment.
I feel like these classes should be refactored. It seems like 90% of the code here is the same, couldn't you change ITrialRunner into an abstract base class that does all the shared work and the actual implementations only have to do the remaining small amount? #Resolved
There was a problem hiding this comment.
I hardly think so. Although 90% of the code looks the same, actually it's not. The code for training a model, evaluating a model and creating training result is different for each scenario and each dataset setting, so the benefits of pulling out all trial runners's shared code into a base class is small considering that the training/evaluation/creating result code still need to be somewhere. In this case I'd prefer to not having a common class for those trial runners in this case.
| #nullable enable | ||
| namespace Microsoft.ML.AutoML | ||
| { | ||
| internal interface ITrialRunnerFactory |
There was a problem hiding this comment.
It's just here for now, But I plan to add another class which would also implement this interface. The code will look like this
internal class CustomTrialRunnerFactory: ITrialRunnerFactory
{
public class CustomTrialRunnerFactory(ITrialRunner instance)
{
this.instance = instance;
}
public ITrialRunner? CreateTrialRunner(TrialSettings settings)
{
return this.instance;
}
}The purpose of this class is to provide the ability for users to pass their own ITrialRunner into AutoMLExperiment. Meanwhile this pattern will also be used to increase extensibility for other components in AutoMLExperiment as well.
|
|
||
| namespace Microsoft.ML.AutoML | ||
| { | ||
| internal class Flow2 |
There was a problem hiding this comment.
It's an implementation of an algo in this paper
I can put a url in code to point to that paper, would that be enough
|
|
||
| namespace Microsoft.ML.AutoML | ||
| { | ||
| public class ArrayMath |
There was a problem hiding this comment.
Nope, they are more used as a helper method for array-like numeric structure and efficiency is not the top-priority here. Beside I just find out that most of these methods are not used anywhere so just did a few clean up as well.
|
|
||
| namespace Microsoft.ML.AutoML.Test | ||
| { | ||
| public class TunerTests : BaseTestClass |
There was a problem hiding this comment.
The tests for cfo tuner is migrated from model builder repo and I will put more tests for other tuners and entire automl experiment soon in a separate PR.
|
Command 'run
In' is not supported by Azure Pipelines.
See additional documentation. In reply to: 1103144497 |
|
Command 'run
In' is not supported by Azure Pipelines.
See additional documentation. |
…ttleLittleCloud/machinelearning into u/xiaoyun/addAutoMLExperiment
|
/azp run |
|
Azure Pipelines successfully started running 2 pipeline(s). |
ghost
locked as resolved and limited conversation to collaborators
2 participants
We are excited to review your PR.
So we can do the best job, please check:
Fixes #nnnnin your description to cause GitHub to automatically close the issue(s) when your PR is merged.#6118
This PR implements AutoML Experiment API proposed in #6118. It also includes CFO tuner from model builder, which provides better searching algorithm.
In implementation,
AutoMLExperimentclass is essentially a service collection forITuner,IRunnerand other components which can be replaced or expanded easily.