Exception has been thrown by the target of an invocation.

[Symbai]

System Information (please complete the following information):

• Model Builder Version (available in Manage Extensions dialog):
  

• Visual Studio Version 17.7.0 Preview 2.0

Describe the bug

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelOperationsCatalog.Load(Stream stream, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.SetTorchRunTimeFolderAndLoadModel(ITrainingConfiguration configuration, String modelPath, MLContext& context, ITransformer& model, DataViewSchema& inputSchema) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 139
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.GenerateConsumptionAsync(ITrainingConfiguration configuration, String nameSpace, String className, TargetType target, String[] labels, CancellationToken ct) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 154   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.ML.ModelBuilder.ViewModels.TrainViewModel.<GenerateCodeBehindFilesAsync>d__100.MoveNext()

To Reproduce

1. I created a console project using .NET 7
2. I added a ML scenario for "Text Classification"
3. I added my CSV data (only 10 rows)
4. I selected the columns
5. I clicked on "Train"
6. A popup appears showing me the training time etc. looks all good
7. Exception occurs

Expected behavior
I expect to proceed to "Evaluate" step. If I manually go to this tab and press the "Predict" button, I get the same exception as above.

Screenshots

Additional context
Last log entry in VS is Generate code behind files

[LittleLittleCloud]

Are you training on GPU, and would you mind sharing your mbconfig

[Symbai]

I was testing both

mbconfig:

{
  "Scenario": "TextClassification",
  "DataSource": {
    "Type": "TabularFile",
    "Version": 1,
    "FilePath": "C:\\Users\\User\\Desktop\\test.csv",
    "Delimiter": ",",
    "DecimalMarker": ".",
    "HasHeader": false,
    "ColumnProperties": [
      {
        "ColumnName": "col0",
        "ColumnPurpose": "Label",
        "ColumnDataFormat": "String",
        "IsCategorical": false,
        "Type": "Column",
        "Version": 4
      },
      {
        "ColumnName": "col1",
        "ColumnPurpose": "Text",
        "ColumnDataFormat": "String",
        "IsCategorical": false,
        "Type": "Column",
        "Version": 4
      }
    ]
  },
  "Environment": {
    "Type": "LocalCPU",
    "Version": 1
  },
  "RunHistory": {
    "Version": 3,
    "Type": "Result",
    "Trials": [
      {
        "Version": 1,
        "Type": "Trial",
        "TrainerName": "TextClassificationMulti",
        "Score": 0.1,
        "RuntimeInSeconds": 5.194,
        "Parameter": {
          "_SCHEMA_": "e0 * e1 * e2",
          "e0": {
            "OutputColumnName": "col0",
            "InputColumnName": "col0",
            "AddKeyValueAnnotationsAsText": false
          },
          "e1": {
            "LabelColumnName": "col0",
            "Sentence1ColumnName": "col1",
            "ScoreColumnName": "Score",
            "OutputColumnName": "PredictedLabel",
            "BatchSize": 32,
            "MaxEpochs": 10,
            "Architecture": "Roberta"
          },
          "e2": {
            "OutputColumnName": "PredictedLabel",
            "InputColumnName": "PredictedLabel"
          }
        }
      }
    ],
    "Estimators": {
      "e0": "MapValueToKey",
      "e1": "TextClassificationMulti",
      "e2": "MapKeyToValue"
    },
    "Schema": "e0 * e1 * e2",
    "MetricName": "MacroAccuracy",
    "ModelFilePath": "C:\\Users\\User\\source\\repos\\MlTest\\MlTest\\MLModel2.mlnet"
  },
  "Type": "TrainingConfig",
  "Version": 4,
  "TrainingOption": {
    "Version": 1,
    "Type": "TextClassficationTrainingOption",
    "TrainingTime": 2147483647,
    "LabelColumn": "col0",
    "TextColumn": "col1",
    "ValidationOption": {
      "Version": 0,
      "Type": "CrossValidationValidationOption",
      "NumberOfFolds": 5
    }
  }
}

bandicam.2023-06-16.05-29-19-286.mp4

[LittleLittleCloud]

Yea please. Thanks!

[daikoz]

same issue with same scenario with:
ML.NET version 17.17.0.2332602
Microsoft Visual Studio 2022 Version 17.6.4

[Symbai]

I am just experiencing it again on a different project. All I did was choosing a new TSV file and clicked on 'Train again'. But I also created a brand new project and got the same immediately on first train. Here is some updated info:

Exception:

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelOperationsCatalog.Load(Stream stream, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.SetTorchRunTimeFolderAndLoadModel(ITrainingConfiguration configuration, String modelPath, MLContext& context, ITransformer& model, DataViewSchema& inputSchema) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 139
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.GenerateConsumptionAsync(ITrainingConfiguration configuration, String trainingConfigurationFolder, String nameSpace, String className, TargetType target, String[] labels, CancellationToken ct) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 155   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.ML.ModelBuilder.ViewModels.TrainViewModel.<GenerateCodeBehindFilesAsync>d__100.MoveNext()

VS Log:

Set log file path to C:\Users\XXX\AppData\Local\Temp\MLVSTools\logs\MLModel1-YC9ZBU.txt
start text classification 
restore "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj" --configfile "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\NuGet.config" -r win-x64 /p:UsingToolXliff=false /p:TorchSharpVersion=0.99.5 /p:TorchSharpCudaRuntimeVersion=1.13.0.1 /p:TensorflowRuntimeVersion=2.3.1 /p:BaseIntermediateOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\obj"
publish "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj" -r win-x64 -c Release --no-self-contained -o "C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5" --no-restore /p:UsingToolXliff=false /p:TorchSharpVersion=0.99.5 /p:TorchSharpCudaRuntimeVersion=1.13.0.1 /p:TensorflowRuntimeVersion=2.3.1 /p:BaseOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\bin\\" /p:BaseIntermediateOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\obj\\"
start installing runtime in C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5
  Determining projects to restore...
  Restored C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj (in 418 ms).
MSBuild version 17.8.0-preview-23418-03+0125fc9fb for .NET
C:\Program Files\dotnet\sdk\8.0.100-rc.1.23455.8\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(311,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj]
  torchsharp.gpu -> C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\bin\Release\netstandard2.0\win-x64\torchsharp.gpu.dll
  torchsharp.gpu -> C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\
install runtime successfully
Use cross validation with fold: 5
|      Trainer                             MacroAccuracy Duration    |
|--------------------------------------------------------------------|
|0     TextClassificationMulti             0,0102     463,1580       |
|--------------------------------------------------------------------|
|                          Experiment Results                        |
|--------------------------------------------------------------------|
|                               Summary                              |
|--------------------------------------------------------------------|
|ML Task: text classification                                        |
|Dataset: H:\xxxx.tsv                                        |
|Label : col0                                                        |
|Total experiment time :   463,1580 Secs                             |
|Total number of models explored: 1                                  |
|--------------------------------------------------------------------|
|                        Top 1 models explored                       |
|--------------------------------------------------------------------|
|      Trainer                             MacroAccuracy Duration    |
|--------------------------------------------------------------------|
|0     TextClassificationMulti             0,0102     463,1580       |
|--------------------------------------------------------------------|
Generate code behind files

Log file:

(Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update TrainViewModel_163.TrainingDidNotSucceed (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.TryOutRow (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.TopFiveRecommendationLabel (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.IsTryOutItemIdValueInDataset (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.IsTryOutUserIdValueInDataset (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update ForecastingTryOutViewModel_197.IsShowTable (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update ForecastingTryOutViewModel_197.IsShowGraph (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0973 DEBUG update ForecastingTryOutViewModel_197.IsShowTable (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0973 DEBUG update ForecastingTryOutViewModel_197.IsShowGraph (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG fail calling update handler GetSampleCodeSnippetAsync (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG Exception has been thrown by the target of an invocation. (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG    at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ML.ModelBuilder.ToolWindows.ConsumeViewModel.<GetSampleCodeSnippetAsync>d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ML.ModelBuilder.ViewModel.CodeGenerator.DependencyGraphManager.<>c__DisplayClass15_0.<<Initialize>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.ML.ModelBuilder.ViewModel.CodeGenerator.DependencyGraphManager.<TryExecuteNotifyDependencyNodeDelegateAsync>d__20.MoveNext() (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)

[LittleLittleCloud]

Looks like you are running with cross-validation? The fix for cross-validation hasn't been fix yet so you might need to use another validation strategy

[Symbai]

Unbelievable, you're right. I switched from cross-validation (which seems to be the default option because I never changed that) to split-validation and this fixed it. But cross-validation seems to be the better one, any ETA when it will be fixed? Or can you at least say if the team is working on it?

//edit: With the first run using split-validation the results were acceptable. Now I've trained again and I get completely different prediction results comparing to the first run. Not just slightly different values but completely(!) different results. Even though the data which is used for training hasn't changed.

[LittleLittleCloud]

@Symbai Actually, looks like the issue has already been fixed by me,,,, can you try upgrade model builder to v17.17.* and give it another trial?

[Symbai]

The VS extension I have says its the latest version: 17.17.0.2332602 see screenshot in my previous post #2716 (comment)

//edit: Reinstalled the extension but still the same

[LittleLittleCloud]

OK, let me double check then