[C#] Should support StructField arrays

[asfimport]

The C# implementation of Arrow does not support struct arrays and, complex types more generally.

 I notice ARROW-6870 addresses Dictionary arrays however this is not as flexible as structs (for example, cannot mix data types)

The source does have a stub for StructArray however there is no Builder nor example on how to use it so I can assume it is not supported.

 

Reporter: Cameron Murray
Assignee: Prashanth Govindarajan / @pgovind

PRs and other links:

• GitHub Pull Request #8348

_{Note: This issue was originally created as ARROW-6972. Please see the migration documentation for further details.}

[asfimport]

Steve Suh:
We are currently trying to support Spark 3.0 in .NET for Apache Spark . We are tracking it with issue 654. To support this, we need to be able to create a StructArray with StructType/Fields. However, when attempting to create these, we run into the following exception:

ProcessStream() failed with exception: System.NotImplementedException: The method or operation is not implemented.
   at Apache.Arrow.Ipc.ArrowTypeFlatbufferBuilder.TypeVisitor.Visit(IArrowType type)
   at Apache.Arrow.Types.ArrowType.Accept[T](T type, IArrowTypeVisitor visitor)
   at Apache.Arrow.Types.StructType.Accept(IArrowTypeVisitor visitor)
   at Apache.Arrow.Ipc.ArrowStreamWriter.SerializeSchema(Schema schema)
   at Apache.Arrow.Ipc.ArrowStreamWriter.WriteSchemaAsync(Schema schema, CancellationToken cancellationToken)
   at Apache.Arrow.Ipc.ArrowStreamWriter.WriteRecordBatchInternalAsync(RecordBatch recordBatch, CancellationToken cancellationToken)
   at Microsoft.Spark.Worker.Command.ArrowOrDataFrameGroupedMapCommandExecutor.ExecuteArrowGroupedMapCommand(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\Source\Repos\github\suhsteve\spark\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 770
   at Microsoft.Spark.Worker.Command.ArrowOrDataFrameGroupedMapCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\Source\Repos\github\suhsteve\spark\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 740
   at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\Source\Repos\github\suhsteve\spark\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 75
   at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\Source\Repos\github\suhsteve\spark\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 57
   at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\Source\Repos\github\suhsteve\spark\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 153

[asfimport]

Eric Erhardt / @eerhardt:
To unblock .NET for Apache Spark, we need to get this fix into 2.0. Marking it as a blocker so we can get this change in before 2.0 ships.

[asfimport]

Neal Richardson / @nealrichardson:
Hi @eerhardt, as discussed on the mailing list, we're looking to do the 2.0 release on October 9. Will you be able to get this issue resolved in the next week?

[asfimport]

Prashanth Govindarajan / @pgovind:
Hello @nealrichardson. I believe so, yes. I have it working locally already, I just need to do some minor debugging and cleanup.

[asfimport]

Prashanth Govindarajan / @pgovind:
Opened https://github.com/apache/arrow/pull/8348/files for this

[asfimport]

Eric Erhardt / @eerhardt:
Issue resolved by pull request 8348
#8348