ARROW-5494: [Python] Create FileSystem bindings

[kszucs]

• FileSystem binding without S3 (it will be implemented in a followup PR)
• Moved arrow/python/util/datetime.h one level upper, enabling to be used by the bindings

[kszucs]

@pitrou apply ARROW_PYTHON_EXPORT for all?

[pitrou]

Well, the functions which were inlined should be kept inline IMO.

[kszucs]

Then I need to move all of the implementation to the header, including the static functions (practically the whole implementation).

[pitrou]

That's how it was already, right?

[kszucs]

If we keep everything as internal than that could work, although I prefer readable header files.
Actually we should benchmark whether it has a performance impact or not.

Could you recommend me specific benchmarks if we have any?

[pitrou]

We don't seem to have any datetime-heavy benchmarks in the ASV benchmarks directory.
However, you can probably devise a timeit-based microbenchmark easily. No need to be sophisticated :-)

[kszucs]

@wesm how can I properly set the visibility for https://github.com/apache/arrow/pull/5258/files#diff-6de8e16eea663cbb2694f9e66e71bd75R55 ?

[wesm]

@kszucs it's a typedef... visibility only has to do with exported compiled symbols

[kszucs]

Ahh, indeed! Thanks!

[kszucs]

@pitrou I have a "not a subpath error" which is probably windows related? https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/27224120/job/dx9qmxdj5tnenu94#L2057

[pitrou]

@pitrou I have a "not a subpath error" which is probably windows related?

Right now the filesystem classes expect forward slashes, and it seems you're passing backslashes. You should try pathlib.Path.as_posix.

[pitrou]

Thanks for doing this. It's gonna be very useful. I posted some comments below.

[pitrou]

We don't seem to have any datetime-heavy benchmarks in the ASV benchmarks directory.
However, you can probably devise a timeit-based microbenchmark easily. No need to be sophisticated :-)

[pitrou]

Looks like the AppVeyor failure needs fixing, can you take a look @kszucs ?

[jorisvandenbossche]

I quickly tried this out, and added a few comments. But looks good!

[jorisvandenbossche]

For the pathlib comment, the other alternative would also be to add it as a dependency (for python 2 only, just like also enum34 is a py2 only dependency for enum backport).
I would say that if we think it is nice to return a pathlib, we shouldn't preclude ourselves doing that just for the few months of py2 support we still have.

[kszucs]

After transitioning to py3 only it would be reasonable to use pathlib.Path objects as return types indeed (at least that's my preference). I suggest to create a JIRA where we can discuss further (possible involves other parts of pyarrow), meanwhile I've set the return types to string.

[pitrou]

After transitioning to py3 only it would be reasonable to use pathlib.Path objects as return types indeed (at least that's my preference).

You mean from get_target_stats?

[pitrou]

it would be reasonable to use pathlib.Path objects as return types indeed (at least that's my preference)

Actually, no. pathlib paths are only for the local filesystem, not for arbitrary resources (e.g. S3). Instead you should return a plain string.

[jorisvandenbossche]

Actually, no. pathlib paths are only for the local filesystem, not for arbitrary resources (e.g. S3). Instead you should return a plain string.

Ah, that solves the python 2 / should we depend on pathlib question then.

[kszucs]

We could have a pathlib compatible Path implementation as well. Whatever, the API we're providing is not stable yet.

[pitrou]

For now I'm not doing a full review again, feel free to ping me when you think you're ready (and CI is fixed :-)).

[kszucs]

@pitrou this should be green now. I'd like to move forward with bindings for S3 once it is merged.

[pitrou]

Just two more comments, sorry ;-)

[pitrou]

I still think this shouldn't accept Path objects, only str and bytes. Path objects are only valid for local paths, but here we are dealing with paths in arbitrary systems.

[kszucs]

@pitrou done, actually we don't accept bytes neither, only strings (we're discussing it previously)

[kszucs]

@pitrou done, builds are green, the travis error is about codecov upload failure.

[pitrou]

This is a very nice start, thank you :-)