ARROW-16782: [Format] Add REE definitions to FlatBuffers

[zagto]

No description provided.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-16782

[alamb]

FBS change looks good to me -- thank you @zagto

Is it possible to incorporate the documentation from #13333 into this pR as well so that the new changes are documented?

For anyone following along, see https://lists.apache.org/thread/539scy67qom5t2fkkd1m6fvh5htvwo3s discussion for more context

cc @zeroshade

[pitrou]

Is it possible to incorporate the documentation from #13333 into this pR as well so that the new changes are documented?

+1 on this. It will also make reviewing easier.

[zeroshade]

@alamb @pitrou I rebased the docs from #13333 into this PR (to maintain the commit history from @zagto) and then updated them with the new name being "Run-End encoded" rather than "Run-Length encoded". Let me know if there's anything else needed to add.

[zeroshade]

@github-actions crossbow submit preview-docs

EDIT: this PR was old, I've rebased it now and am going to make this comment again so it should work now.

[github-actions]

Unable to match any tasks for `preview-docs`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/3840303309

[zeroshade]

@westonpace I've updated with a line prescribing the standard names for the children and updated the example layout to add the underscore so that it uses those standard names run_ends and values.

Once I get some explicit approvals on this I'll merge it in.

[lidavidm]

General question here: how are we going to handle versioning of the Arrow specification? Are we going to merge this and down the line release an "Arrow specification 1.1.0"/MetadataVersion.V6 once we also have types like StringView, or are we going to bump these versions for each new type?

[zeroshade]

That is a very good question @lidavidm.

What do you think @westonpace @pitrou @alamb @ianmcook ?

[pitrou]

I think we only increase MetadataVersion when we make incompatible changes to the Flatbuffers definition, which is not the case here.

As for the format version, we should perhaps bump it:

/// Format Version History.
/// Version 1.0 - Forward and backwards compatibility guaranteed.
/// Version 1.1 - Add Decimal256 (No format release).
/// Version 1.2 (Pending)- Add Interval MONTH_DAY_NANO

[zeroshade]

@pitrou I can bump it in the docs in this PR, but do we need to do a vote on the mailing list or anything to make the new version official?

[pitrou]

I don't think a new vote is needed, though I don't know for sure.

[lidavidm]

I suppose the version bump is implicit in adding a new data type? Looking at the Git history/the MONTH_DAY_NANO PR, the versions were just bumped when we added the type. (...And 1.2 is still "pending", guess we should take that out now and put 1.3 as adding REE.)

[zeroshade]

I've updated the Schema.fbs and Columnar.rst files for the version and regenerated files to pick up the new comments.

I've got two approvals here, so unless someone comments otherwise I'll merge this after EOD today, to give people a chance to add any final comments. Thanks everyone!

[alamb]

I reviewed only the changes to docs/source/format/Columnar.rst and format/Schema.fbs but they look great to me. Thank you to everyone who has been working on this proposal -- it is super exciting to see it so close

[alamb]

❤️

[ursabot]

Benchmark runs are scheduled for baseline = d0ffedb and contender = 3146588. 3146588 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.99% ⬆️0.24%] test-mac-arm
[Finished ⬇️0.51% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.84% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 3146588a ec2-t3-xlarge-us-east-2
[Finished] 3146588a test-mac-arm
[Finished] 3146588a ursa-i9-9960x
[Finished] 3146588a ursa-thinkcentre-m75q
[Finished] d0ffedbe ec2-t3-xlarge-us-east-2
[Failed] d0ffedbe test-mac-arm
[Finished] d0ffedbe ursa-i9-9960x
[Finished] d0ffedbe ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java