Read "arrow" (IPC and streaming) files usning org.apache.arrow.dataset.jni.NativeDatasetFactory in Java API

[igor-suhorukov]

How to fetch "arrow" (IPC and streaming) files usning org.apache.arrow.dataset.jni.NativeDatasetFactory in Java API. This chapter is missing in documentation. It is possible to do in CPP/Python API

[lwhite1]

Hi @igor-suhorukov,

I would need to check, but I don't think this is implemented in Java Dataset yet.

Do you need to use Datasets for your application? (in other words, is your data spread over multiple files or too big to fit in memory?) If not, you should be able to use a FileReader load the data.

[igor-suhorukov]

This functionality required to implement Arrow file/Stream input format in my use case to process large amount of existing geospatial ARROW format data in Apache Spark data source. Optimized Analytics Package (OAP) for Spark also can leverage this feature of Dataset on JVM. They use FileSystemDatasetFactory in this Spark gazelle_plugin adapter .

Jira ticket for this improvement.