[C++] The C++ API for writing datasets could be improved

[asfimport]

I was working on write dataset testing in the C++ API today and ran into a number of things that were not very intuitive. All of these are abstracted away / hidden by the python / R interface so this really only applies to anyone using the C++ API directly.

• If no partitioning is specified the write will segfault. Instead it should us a default (no-op) partitioning.
• The min_rows_per_group option should probably default to something higher than 0
• It's not clear how to specify the format (you do it by creating a format, then setting the file write options, which sets the format privately)
• There is no default for basename_template
• There is no default for filesystem (should be local filesystem)

Reporter: Weston Pace / @westonpace

Related issues:

• [C++] Improve Dataset Write Option Defaults (is duplicated by)

PRs and other links:

• GitHub Pull Request #13959

_{Note: This issue was originally created as ARROW-15409. Please see the migration documentation for further details.}

[asfimport]

Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

[vibhatha]

@westonpace there is an issue in defining the format. Because all the format definitions depends on the definition on this file_base.h base class. It is not very clear how to set this value.

Also note that looking into the previous PR: #13959

If min_rows_per_group value is set to a higher default value and if we set the max_rows_per_group, and in an event of max_rows_per_group < min_rows_per_group, it fails an existing assertion which make sense. So even in a simpler case where user just needs to set the max value, they may be forced to set the min value.

So I would say keeping it 0 as default would be okay. Am I missing any thing here?

Appreciate your thoughts on these two points.

[westonpace]

@westonpace there is an issue in defining the format. Because all the format definitions depends on the definition on this file_base.h base class. It is not very clear how to set this value.

Make the format a required constructor property. Get rid of the no-arg constructor.

So I would say keeping it 0 as default would be okay. Am I missing any thing here?

A default of 1Mi for min_rows_per_group and 64Mi for max_rows_per_group should be very reasonable and give better performance than leaving it at 0 in many cases.

[kou]

If no partitioning is specified the write will segfault. Instead it should us a default (no-op) partitioning.

This will be resolved by #15256.

[vibhatha]

Should we close this. @kou

[kou]

I don't think so because at least #15256 doesn't solve "There is no default for filesystem (should be local filesystem)".

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.