[R] Support for `connection` class when reading and writing files

[asfimport]

We have an internal filesystem that we interact with through objects that inherit from the connection class. These files aren't necessarily local, making it slightly more complicated to read and write parquet files, for example.

For now, we're generating raw vectors and using that to create the file. For example, to read files

ReadParquet <- function(filename, ...) {}}
   file <-file(filename,"rb")
   on.exit(close(file))
   raw <- readBin(file, "raw", FileInfo(filename)$size)
   return(arrow::read_parquet(raw, ...))
}

And to write,

WriteParquet <- function(df, filepath, ...) {
   stream <- BufferOutputStream$create()
   write_parquet(df, stream, ...)
   raw <- stream$finish()$data()
   file <- file(filepath, "wb")
   on.exit(close(file)
   writeBin(raw, file)
   return(invisible())
}

At the C++ level, we are interacting with R_new_custom_connection defined here:
https://github.com/wch/r-source/blob/trunk/src/include/R_ext/Connections.h

I've been very impressed with how feature-rich arrow is. It would be nice to see this API supported as well.

Reporter: Michael Quinn
Assignee: Dewey Dunnington / @paleolimbot

Related issues:

• [R] Stream reader/writer API that takes socket stream (is duplicated by)

PRs and other links:

• GitHub Pull Request #12323

_{Note: This issue was originally created as ARROW-9235. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
Sounds similar to ARROW-4512.

Would you be interested in submitting a pull request? Seems like you're familiar with the terrain.

[asfimport]

Michael Quinn:
Yeah. It's fundamentally the same issue. They are both instances of R's "connection" class.
https://stat.ethz.ch/R-manual/R-devel/library/base/html/connections.html

Unfortunately, I don't really understand enough about arrow to implement something like this.

[asfimport]

Neal Richardson / @nealrichardson:
And I don't understand enough of R_new_custom_connection to implement this either :) But maybe we can put our heads together. I'm happy to help with the arrow bits if you can get something started.

[asfimport]

Neal Richardson / @nealrichardson:
Looks like readr reads from connections by dumping their contents to a temp file and then reading the file: https://github.com/tidyverse/readr/blob/master/src/connection.cpp

[asfimport]

Ian Cook / @ianmcook:
Since Arrow does not yet support HTTP URIs in file reading functions (ARROW-7594), if we do implement support for connections in the R package, we should consider detecting HTTP URI strings and using connections to handle them.

[asfimport]

Dewey Dunnington / @paleolimbot:
I'll take a look at this since I just did an InputStream ticket. Reading and writing from a connection uses a similar API to the current InputStream and OutputStream, so in theory it's not too bad (with the caveat that connections probably aren't safe to be called from other threads).

https://github.com/wch/r-source/blob/trunk/src/include/R_ext/Connections.h#L91-L94

[asfimport]

Neal Richardson / @nealrichardson:
Issue resolved by pull request 12323
#12323