Skip to content

Ignore hidden columns in AutoML schema checks of validation data #4490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
daholste merged 2 commits into from
Nov 21, 2019
Merged

Ignore hidden columns in AutoML schema checks of validation data #4490

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
41e7d36
Ignore hidden columns in AutoML schema checks of validation data
daholste Nov 20, 2019
e24d565
comment rev
daholste Nov 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion src/Microsoft.ML.AutoML/Utils/UserInputValidationUtil.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,14 +183,22 @@ private static void ValidateValidationData(IDataView trainData, IDataView valida

const string schemaMismatchError = "Training data and validation data schemas do not match.";

if (trainData.Schema.Count != validationData.Schema.Count)
if (trainData.Schema.Count(c => !c.IsHidden) != validationData.Schema.Count(c => !c.IsHidden))
{
throw new ArgumentException($"{schemaMismatchError} Train data has '{trainData.Schema.Count}' columns," +
$"and validation data has '{validationData.Schema.Count}' columns.", nameof(validationData));
}

// Validate that every active column in the train data corresponds to an active column in the validation data.
// (Indirectly, since we asserted above that the train and validation data have the same number of active columns, this also
// esnures the reverse -- that every active column in the validation data corresponds to an active column in the train data.)
foreach (var trainCol in trainData.Schema)

@justinormont justinormont Nov 21, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could use a comment...

Suggested change
foreach (var trainCol in trainData.Schema)
// Also indirectly checks for new columns in the validation datasets as we above enforce the column counts are equal
foreach (var trainCol in trainData.Schema)

I was otherwise going to suggest we check the reverse direction. A comment helps future readers to not have to think through it.

{
if (trainCol.IsHidden)
{
continue;
}

var validCol = validationData.Schema.GetColumnOrNull(trainCol.Name);
if (validCol == null)
{
Expand Down
29 changes: 29 additions & 0 deletions test/Microsoft.ML.AutoML.Tests/UserInputValidationTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
// See the LICENSE file in the project root for more information.

using System;
using System.Collections.Generic;
using System.IO;
using Microsoft.ML.Data;
using Xunit;
Expand Down Expand Up @@ -307,6 +308,34 @@ public void ValidateEmptyValidationDataThrows()
}
}


[Fact]
public void TestValidationDataSchemaChecksIgnoreHiddenColumns()
{
var mlContext = new MLContext();

// Build training data where label column is a float.
var trainDataBuilder = new ArrayDataViewBuilder(mlContext);
trainDataBuilder.AddColumn("Number", NumberDataViewType.Single, 0f);
trainDataBuilder.AddColumn(DefaultColumnNames.Label, NumberDataViewType.Single, 0f);
var trainingData = trainDataBuilder.GetDataView();

// In the training data, transform the label column from a float to a Boolean. This has the effect of
// creating a hidden column named 'Label' of type float and an additional column named 'Label' of type Boolean.
var convertLabelToBoolEstimator = mlContext.Transforms.Conversion.MapValue(DefaultColumnNames.Label,
new List<KeyValuePair<float, bool>>() { new KeyValuePair<float, bool>(1, true) });
trainingData = convertLabelToBoolEstimator.Fit(trainingData).Transform(trainingData);

// Build validaiton data where label column is a Boolean.
var validationDataBuilder = new ArrayDataViewBuilder(mlContext);
validationDataBuilder.AddColumn("Number", NumberDataViewType.Single, 0f);
validationDataBuilder.AddColumn(DefaultColumnNames.Label, BooleanDataViewType.Instance, false);
var validationData = validationDataBuilder.GetDataView();

UserInputValidationUtil.ValidateExperimentExecuteArgs(trainingData, new ColumnInformation(), validationData, TaskKind.BinaryClassification);
}


private static void ValidateLabelTypeTestCore<LabelRawType>(TaskKind task, PrimitiveDataViewType labelType, bool labelTypeShouldBeValid)
{
var dataViewBuilder = new ArrayDataViewBuilder(new MLContext());
Expand Down