GH-35084: [Docs][Format] Add how to change format specification by kou · Pull Request #35174 · apache/arrow

kou · 2023-04-17T07:22:54Z

Rationale for this change

It's for easy to refer.

What changes are included in this PR?

This is based on the discussion:

[DISCUSS] Format changes: process and requirements
https://lists.apache.org/thread/9t0pglrvxjhrt4r4xcsc1zmgmbtr8pxj

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

Closes: [Docs][Format] Add how to change format specification #35084

kou · 2023-04-17T07:23:13Z

@github-actions crossbow submit preview-docs

github-actions · 2023-04-17T07:23:14Z

Closes: [Docs][Format] Add how to change format specification #35084

github-actions · 2023-04-17T07:23:16Z

⚠️ GitHub issue #35084 has been automatically assigned in GitHub to PR creator.

github-actions · 2023-04-17T07:25:37Z

Revision: e3f9af0fa3ad7460a6146e506cc0ce8e5732b51c

Submitted crossbow builds: ursacomputing/crossbow @ actions-2656904870

Task	Status
preview-docs

yoshimotoyuk · 2023-04-17T07:57:47Z

Suggested change

DISCUSS process is a process to discuss the format changes in

DISCUSS process is a process to discuss capability of the format changes in

Discussion is to gather opinion about the new format change suggestion, and see whether if it is capable to be adopted.

yoshimotoyuk · 2023-04-17T08:09:41Z

Suggested change

VOTE process is a process to tell whether we have reached

consensus. We can start a vote for the format changes after we reach

VOTE process is a process to poll whether we have decided to adapt changes or not.

We can start a vote for the format changes after we reach

Current text only tells that when the VOTE starts, there are consensus. But it is happening when DISCUSS reach the consensus. And I believe the VOTE process is the action to finalize the adoption. So I tried to text it that way.

yoshimotoyuk

I have read through the texts.

Few sentence seems missing detail information. So I have added the changes to be more descriptive.

pitrou · 2023-04-17T15:37:43Z

cc @wjones127 for wording.

wgtmac · 2023-04-17T15:41:37Z

Does this follow the same rule of apache voting? For example, which vote is considered binding?

Yes.
"[VOTE][Format] Fixed shape tensor Canonical Extension Type" https://lists.apache.org/thread/f303yscj8gq4oggvvcbvf5zpo972l1kk is a recent example.

wgtmac · 2023-04-17T15:48:41Z

How do we define a reference implementation is complete? For example, if a new data/array type is proposed, does the IPC, compute functions, and parquet module should be implemented as well?

I don't think we need to lay down hard rules. Below we link to https://arrow.apache.org/docs/status.html, which gives us a grid of implementations and their status. Which parts are relevant will depend on the feature IMO. For most new data types, I suspect what's important is IPC implementation.

It's ~up to the voters, effectively. I think it'd be hard to pin this down in text.

That said, this part is discussing whether the implementation is considered a reference Arrow implementation. Which is more about whether the the implementation is self-contained (as opposed to a binding) and ~completeness of the Arrow specification and development activity.

wjones127

I think we can call the process just "discussion and voting", and mention the [VOTE] and [DISCUSS] tags where appropriate.

wjones127 · 2023-04-17T16:30:33Z

I don't think we need to lay down hard rules. Below we link to https://arrow.apache.org/docs/status.html, which gives us a grid of implementations and their status. Which parts are relevant will depend on the feature IMO. For most new data types, I suspect what's important is IPC implementation.

westonpace

Thanks for writing this up.

kou · 2023-04-18T07:38:29Z

@yoshimotoyuk @wjones127 @westonpace Thanks for your suggestions!
I've merged them.

wjones127

Two minor changes. Otherwise this looks good. Thanks for adding this @kou

kou · 2023-04-18T20:24:12Z

Thanks! I've merged them.
I'll start a vote for this later before we merge this.

kou · 2023-04-18T20:24:39Z

@github-actions crossbow submit preview-docs

This is based on the discussion: [DISCUSS] Format changes: process and requirements https://lists.apache.org/thread/9t0pglrvxjhrt4r4xcsc1zmgmbtr8pxj

Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Weston Pace <weston.pace@gmail.com>

Co-authored-by: Will Jones <willjones127@gmail.com>

kou · 2023-04-19T02:31:44Z

@github-actions crossbow submit preview-docs

kou · 2023-04-19T05:18:11Z

@github-actions crossbow submit preview-docs

github-actions · 2023-04-19T05:20:39Z

Revision: a21b227

Submitted crossbow builds: ursacomputing/crossbow @ actions-ad4f16d624

Task	Status
preview-docs

kou · 2023-04-19T07:19:03Z

Preview: http://crossbow.voltrondata.com/pr_docs/35174/format/Changing.html

kou · 2023-04-24T02:52:51Z

The vote thread: https://lists.apache.org/thread/jlc4wtt09rfszlzqdl55vrc4dxzscr4c

pitrou · 2023-04-24T08:59:11Z

Should something be added about documentation as well? Or is it implicit?

kou · 2023-04-24T09:48:43Z

I'm sorry but I can't understand what you intent.
What documentation are you referring?
Design documentation like https://docs.google.com/document/d/1jhPyPZSOo2iy0LqIJVUs9KWPyFULVFJXTILDfkadx2g/edit?usp=sharing for https://lists.apache.org/thread/247z3t06mf132nocngc1jkp3oqglz7jp ?

pitrou · 2023-04-24T09:56:09Z

What documentation are you referring?

Everything under https://arrow.apache.org/docs/format/

alamb

Looks good to me. I agree it might be nice to explicitly add something about updating the documentation as part of the format change, I can also see the argument for that being implicit.

kou · 2023-04-26T00:17:11Z

Ah, I understand.
(Thanks for your comment @alamb!)

OK. I've add a note for it:

.. note::

   We must update the corresponding documentation (files in
   `<https://github.com/apache/arrow/tree/main/docs/source/format>`_)
   too.

If anyone has a suggestion for wording, I appreciate it. (I'm not a native English speaker.)

kou · 2023-04-26T00:17:19Z

@github-actions crossbow submit preview-docs

github-actions · 2023-04-26T00:19:39Z

Revision: dbd2c20

Submitted crossbow builds: ursacomputing/crossbow @ actions-026a10f6d6

Task	Status
preview-docs

kou · 2023-04-27T07:01:17Z

The vote result: https://lists.apache.org/thread/ophn3b58gb4rmjow00d3zovox7vnhod2

I'll merge this.

…apache#35174) ### Rationale for this change It's for easy to refer. ### What changes are included in this PR? This is based on the discussion: [DISCUSS] Format changes: process and requirements https://lists.apache.org/thread/9t0pglrvxjhrt4r4xcsc1zmgmbtr8pxj ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * Closes: apache#35084 Lead-authored-by: Sutou Kouhei <kou@clear-code.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Weston Pace <weston.pace@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

github-actions Bot added Component: Documentation awaiting committer review Awaiting committer review labels Apr 17, 2023

yoshimotoyuk reviewed Apr 17, 2023

View reviewed changes

Comment thread docs/source/format/Changing.rst Outdated

yoshimotoyuk reviewed Apr 17, 2023

View reviewed changes

wgtmac reviewed Apr 17, 2023

View reviewed changes

wjones127 requested changes Apr 17, 2023

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Apr 17, 2023

westonpace reviewed Apr 17, 2023

View reviewed changes

Comment thread docs/source/format/Changing.rst Outdated

Comment thread docs/source/format/Changing.rst Outdated

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Apr 18, 2023

wjones127 reviewed Apr 18, 2023

View reviewed changes

Comment thread docs/source/format/Changing.rst Outdated

wjones127 reviewed Apr 18, 2023

View reviewed changes

Comment thread docs/source/format/Changing.rst Outdated

wjones127 approved these changes Apr 18, 2023

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes awaiting merge Awaiting merge and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Apr 18, 2023

kou force-pushed the docs-how-to-change-format-spec branch from 642f6e5 to 784be05 Compare April 18, 2023 20:22

This comment was marked as outdated.

Sign in to view

kou and others added 4 commits April 19, 2023 11:31

apacheGH-35084: [Docs][Format] Add how to change format specification

a0d35a8

This is based on the discussion: [DISCUSS] Format changes: process and requirements https://lists.apache.org/thread/9t0pglrvxjhrt4r4xcsc1zmgmbtr8pxj

Apply suggestions from code review

456b85f

Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Weston Pace <weston.pace@gmail.com>

Apply suggestions from code review

1fcd7fa

Co-authored-by: Will Jones <willjones127@gmail.com>

Fold and remove trailing spaces

8693dd2

kou force-pushed the docs-how-to-change-format-spec branch from 784be05 to 8693dd2 Compare April 19, 2023 02:31

This comment was marked as outdated.

Sign in to view

kou added 2 commits April 19, 2023 14:15

Fix markup

1ad1771

Fix markup

a21b227

lidavidm approved these changes Apr 19, 2023

View reviewed changes

alamb approved these changes Apr 24, 2023

View reviewed changes

kou added 2 commits April 26, 2023 09:12

Add a task about documentation explicitly

474a023

Use note

dbd2c20

kou merged commit 07d02d6 into apache:main Apr 27, 2023

kou deleted the docs-how-to-change-format-spec branch April 27, 2023 07:01

	DISCUSS process is a process to discuss the format changes in
	DISCUSS process is a process to discuss capability of the format changes in

-VOTE process is a process to tell whether we have reached
-consensus. We can start a vote for the format changes after we reach
+VOTE process is a process to poll whether we have decided to adapt changes or not.
+We can start a vote for the format changes after we reach

Uh oh!

Conversation

kou commented Apr 17, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

kou commented Apr 17, 2023

Uh oh!

github-actions Bot commented Apr 17, 2023

Uh oh!

github-actions Bot commented Apr 17, 2023

Uh oh!

github-actions Bot commented Apr 17, 2023

Uh oh!

Uh oh!

yoshimotoyuk Apr 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yoshimotoyuk Apr 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yoshimotoyuk left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou commented Apr 17, 2023

Uh oh!

wgtmac Apr 17, 2023

Choose a reason for hiding this comment

Uh oh!

kou Apr 18, 2023

Choose a reason for hiding this comment

Uh oh!

wgtmac Apr 17, 2023

Choose a reason for hiding this comment

Uh oh!

wjones127 Apr 17, 2023

Choose a reason for hiding this comment

Uh oh!

lidavidm Apr 17, 2023

Choose a reason for hiding this comment

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjones127 Apr 17, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kou commented Apr 18, 2023

Uh oh!

Uh oh!

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

kou commented Apr 18, 2023

Uh oh!

kou commented Apr 18, 2023

Uh oh!

This comment was marked as outdated.

kou commented Apr 19, 2023

kou commented Apr 17, 2023 •

edited by github-actions Bot

Loading

yoshimotoyuk Apr 17, 2023 •

edited

Loading

yoshimotoyuk Apr 17, 2023 •

edited

Loading

yoshimotoyuk left a comment •

edited

Loading

kou commented Apr 26, 2023 •

edited

Loading