[CI][Python] verify-rc python nightly builds fail due to missing some flags that were activated with ARROW_PYTHON=ON

[asfimport]

Some of our nightly builds are failing with:

 [ 35%] Building CXX object CMakeFiles/_dataset.dir/_dataset.cpp.o
/arrow/python/build/temp.linux-x86_64-cpython-38/_dataset.cpp:833:10: fatal error: arrow/csv/api.h: No such file or directory
 #include "arrow/csv/api.h"
          ^~~~~~~~~~~~~~~~~
compilation terminated.

I suspect the flags included CSV=ON when building with PYTHON=ON changes here might be related: 53ac2a0

Example of nightly failures:

https://github.com/ursacomputing/crossbow/actions/runs/3135833175/jobs/5091988801

Reporter: Raúl Cumplido / @raulcd
Assignee: Raúl Cumplido / @raulcd

Related issues:

• [C++][Python] Keep and deprecate ARROW_PYTHON CMake option for backward compatibility (duplicates)

_{Note: This issue was originally created as ARROW-17877. Please see the migration documentation for further details.}

[asfimport]

Raúl Cumplido / @raulcd:
@kou what do you think, should we enable individually on the verify-release-candidate.sh and other jobs all the flags that were activated when ARROW_PYTHON=ON in the past if we are building pyarrow too:

if(ARROW_PYTHON)
  set(ARROW_COMPUTE ON)
  set(ARROW_CSV ON)
  set(ARROW_DATASET ON)
  set(ARROW_FILESYSTEM ON)
  set(ARROW_HDFS ON)
  set(ARROW_JSON ON)
endif() 

or should we create some CMake group flag that enables all the requirements for pyarrow?

There are still quite a lot of occurrences of this flag:

$ grep -r "ARROW_PYTHON=ON" | wc -l
22 

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
See also ARROW-17868, where @kou mentioned to add it back but deprecated.

(that still means we should also update all internal usage and docs that use it)

[asfimport]

Kouhei Sutou / @kou:
Yes. I'll add ARROW_PYTHON back but we should not use ARROW_PYTHON=ON in the verify-release-candidate.sh.

Because the ARROW_PYTHON dependencies are inconsistent. For example, they include ARROW_DATASET=ON but it's an optional component (not a required component) in PyArrow. And they don't include all optional components such as ARROW_PARQUET=ON.

I think that CMake presets will be better replacement for ARROW_PYTHON because we can define multiple presets such as features-python-minimum and features-python-maximum. But CMake presets require CMake 3.19 or later...

[asfimport]

Raúl Cumplido / @raulcd:
Ok, perfect. I am closing this one as duplicate as all the work (updating usages and docs) is being done on ARROW-17868. Thanks @kou !

[asfimport]

Raúl Cumplido / @raulcd:
This issue is a duplicate.

[pareshnagendrapaul-git]

missing header file arrow/csv/api.h error when building pyarrow from source. please give a solution.

[kou]

Could you show how to reproduce your problem?
What version of PyArrow are you using?

[pareshnagendrapaul-git]

Try building Pyarrow from source. I'm outside now, I don't remember exactly but probably 8.0.0... Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Sutou Kouhei ***@***.***> Sent: Monday, March 3, 2025 7:07:28 AM To: apache/arrow ***@***.***> Cc: pareshnagendrapaul-git ***@***.***>; Comment ***@***.***> Subject: Re: [apache/arrow] [CI][Python] verify-rc python nightly builds fail due to missing some flags that were activated with ARROW_PYTHON=ON (Issue #33092) Could you show how to reproduce your problem? What version of PyArrow are you using? — Reply to this email directly, view it on GitHub<#33092 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BP4RU3KZ3RLD2ISOXMEDCWL2SOW5RAVCNFSM6AAAAABYE667QGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJTGA3DCNBUGU>. You are receiving this because you commented.Message ID: ***@***.***> [kou]kou left a comment (apache/arrow#33092)<#33092 (comment)> Could you show how to reproduce your problem? What version of PyArrow are you using? — Reply to this email directly, view it on GitHub<#33092 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BP4RU3KZ3RLD2ISOXMEDCWL2SOW5RAVCNFSM6AAAAABYE667QGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJTGA3DCNBUGU>. You are receiving this because you commented.Message ID: ***@***.***>

[kou]

Could you try the latest PyArrow?