Skip to content

ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure#11583

Closed
pitrou wants to merge 3 commits into
apache:masterfrom
pitrou:oss-fuzz-gtest
Closed

ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure#11583
pitrou wants to merge 3 commits into
apache:masterfrom
pitrou:oss-fuzz-gtest

Conversation

@pitrou

@pitrou pitrou commented Nov 1, 2021

Copy link
Copy Markdown
Member

ARROW-14393 downgraded the bundled GTest version to a version that doesn't build on the OSS-Fuzz setup
(and there doesn't seem to be an obvious fix or workaround around those build errors).

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Also, add a Crossbow task for the OSS-Fuzz build check, since the Github Actions cron job failure went unnoticed.

@pitrou

pitrou commented Nov 1, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g cpp

@github-actions

github-actions Bot commented Nov 1, 2021

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@pitrou pitrou changed the title [CI] [WIP] Always compile GTest from source [CI] Always compile GTest from source Nov 1, 2021
@pitrou pitrou marked this pull request as draft November 1, 2021 15:39
@github-actions

github-actions Bot commented Nov 1, 2021

Copy link
Copy Markdown

Revision: 209b1e6382c4fb4bc81e4f6a3e725fe2d1faddb9

Submitted crossbow builds: ursacomputing/crossbow @ actions-1063

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-debian-10-cpp Github Actions
test-debian-11-cpp Github Actions
test-fedora-33-cpp Github Actions
test-ubuntu-18.04-cpp Github Actions
test-ubuntu-18.04-cpp-release Github Actions
test-ubuntu-18.04-cpp-static Github Actions
test-ubuntu-20.04-cpp Github Actions
test-ubuntu-20.04-cpp-14 Github Actions
test-ubuntu-20.04-cpp-17 Github Actions
test-ubuntu-20.04-cpp-bundled Github Actions
test-ubuntu-20.04-cpp-thread-sanitizer Github Actions

@pitrou

pitrou commented Nov 1, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit test-conda-*

@github-actions

github-actions Bot commented Nov 1, 2021

Copy link
Copy Markdown

Revision: 4da0482af9451acbbd5c7efef6d83040e5d48d97

Submitted crossbow builds: ursacomputing/crossbow @ actions-1064

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou

pitrou commented Nov 1, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit test-conda-*

@pitrou

pitrou commented Nov 1, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g verify-rc-source

@github-actions

github-actions Bot commented Nov 1, 2021

Copy link
Copy Markdown

Revision: 70a5ce2ae1beeac15b2745eaa4a701d4b8b2cb54

Submitted crossbow builds: ursacomputing/crossbow @ actions-1065

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou

pitrou commented Nov 1, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit vcpkg

@github-actions

github-actions Bot commented Nov 1, 2021

Copy link
Copy Markdown

Revision: 70a5ce2ae1beeac15b2745eaa4a701d4b8b2cb54

Submitted crossbow builds: ursacomputing/crossbow @ actions-1066

Task Status
verify-rc-source-cpp-linux-amd64 Github Actions
verify-rc-source-cpp-macos-amd64 Github Actions
verify-rc-source-cpp-macos-arm64 Github Actions
verify-rc-source-csharp-linux-amd64 Github Actions
verify-rc-source-csharp-macos-amd64 Github Actions
verify-rc-source-csharp-macos-arm64 Github Actions
verify-rc-source-go-linux-amd64 Github Actions
verify-rc-source-go-macos-amd64 Github Actions
verify-rc-source-go-macos-arm64 Github Actions
verify-rc-source-integration-linux-amd64 Github Actions
verify-rc-source-integration-macos-amd64 Github Actions
verify-rc-source-integration-macos-arm64 Github Actions
verify-rc-source-java-linux-amd64 Github Actions
verify-rc-source-java-macos-amd64 Github Actions
verify-rc-source-js-linux-amd64 Github Actions
verify-rc-source-js-macos-amd64 Github Actions
verify-rc-source-js-macos-arm64 Github Actions
verify-rc-source-python-linux-amd64 Github Actions
verify-rc-source-python-macos-amd64 Github Actions
verify-rc-source-python-macos-arm64 Github Actions
verify-rc-source-ruby-linux-amd64 Github Actions
verify-rc-source-ruby-macos-amd64 Github Actions
verify-rc-source-ruby-macos-arm64 Github Actions
verify-rc-source-windows Github Actions

@github-actions

github-actions Bot commented Nov 1, 2021

Copy link
Copy Markdown

Revision: 70a5ce2ae1beeac15b2745eaa4a701d4b8b2cb54

Submitted crossbow builds: ursacomputing/crossbow @ actions-1067

Task Status
test-build-vcpkg-win Github Actions

@pitrou

pitrou commented Nov 1, 2021

Copy link
Copy Markdown
Member Author

Hmm, it appears we cannot exercise the release verification script using crossbow (it tries to fetch Arrow 1.0.0)? @kszucs

@pitrou

pitrou commented Nov 2, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit test-conda-*

@github-actions

github-actions Bot commented Nov 2, 2021

Copy link
Copy Markdown

Revision: a9cd09f34907e8183ebcdbca0f44d3145e17b11c

Submitted crossbow builds: ursacomputing/crossbow @ actions-1068

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou

pitrou commented Nov 2, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g fuzz

@pitrou

pitrou commented Nov 2, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g fuzz

@pitrou pitrou changed the title [CI] Always compile GTest from source ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure Nov 2, 2021
@github-actions

github-actions Bot commented Nov 2, 2021

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Nov 2, 2021

Copy link
Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@pitrou

pitrou commented Nov 2, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g fuzz

@github-actions

github-actions Bot commented Nov 2, 2021

Copy link
Copy Markdown

Revision: 7cf7cd67bf5f9748a95787eb0b824bae6d787036

Submitted crossbow builds: ursacomputing/crossbow @ actions-1069

Task Status
test-build-cpp-fuzz Github Actions

@pitrou

pitrou commented Nov 2, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g fuzz

@pitrou

pitrou commented Nov 2, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g fuzz

@github-actions

github-actions Bot commented Nov 2, 2021

Copy link
Copy Markdown

Revision: 860317fead19201a884c7b2029a4ba5d9b8eb62f

Submitted crossbow builds: ursacomputing/crossbow @ actions-1072

Task Status
test-build-cpp-fuzz Github Actions

@pitrou

pitrou commented Nov 2, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g fuzz

@github-actions

github-actions Bot commented Nov 2, 2021

Copy link
Copy Markdown

Revision: 90971770ecc6e54fef48de05e50aa9e8f44c4713

Submitted crossbow builds: ursacomputing/crossbow @ actions-1073

Task Status
test-build-cpp-fuzz Github Actions

ARROW-14393 downgraded the bundled GTest version to a version that doesn't build on the OSS-Fuzz setup
(and there doesn't seem to be an obvious fix or workaround around those build errors).

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Also, add a Crossbow task for the OSS-Fuzz build check, since the Github Actions cron job failure went unnoticed.
@pitrou pitrou marked this pull request as ready for review November 2, 2021 15:52
@pitrou pitrou requested a review from kszucs November 2, 2021 15:52
@pitrou

pitrou commented Nov 2, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit test-conda-*

@github-actions

github-actions Bot commented Nov 2, 2021

Copy link
Copy Markdown

Revision: 0bc448e

Submitted crossbow builds: ursacomputing/crossbow @ actions-1074

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou

pitrou commented Nov 2, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit test-conda-*

@pitrou

pitrou commented Nov 2, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit test-conda-cpp-valgrind

@github-actions

github-actions Bot commented Nov 2, 2021

Copy link
Copy Markdown

Revision: e035ed5

Submitted crossbow builds: ursacomputing/crossbow @ actions-1075

Task Status
test-conda-cpp-valgrind Azure

@pitrou

pitrou commented Nov 3, 2021

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit test-conda-cpp-valgrind

@github-actions

github-actions Bot commented Nov 3, 2021

Copy link
Copy Markdown

Revision: e035ed5

Submitted crossbow builds: ursacomputing/crossbow @ actions-1077

Task Status
test-conda-cpp-valgrind Azure

@pitrou

pitrou commented Nov 3, 2021

Copy link
Copy Markdown
Member Author

@kszucs Can we merge this? I'd like to unblock the OSS-Fuzz builds.

@kszucs

kszucs commented Nov 3, 2021

Copy link
Copy Markdown
Member

Hmm, it appears we cannot exercise the release verification script using crossbow (it tries to fetch Arrow 1.0.0)? @kszucs

We can only build releases with those tasks by providing additional parameters: #11511 (comment)

@kszucs

kszucs commented Nov 3, 2021

Copy link
Copy Markdown
Member

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Do we have a JIRA for that?

@pitrou

pitrou commented Nov 3, 2021

Copy link
Copy Markdown
Member Author

I don't understand the question. What do you want a JIRA for?

@kszucs kszucs closed this in 64b80dd Nov 3, 2021
@pitrou pitrou deleted the oss-fuzz-gtest branch November 3, 2021 15:48
@ursabot

ursabot commented Nov 3, 2021

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 09b79a1 and contender = 64b80dd. 64b80dd is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️1.03% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.36% ⬆️0.04%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

@kszucs

kszucs commented Nov 3, 2021

Copy link
Copy Markdown
Member

I don't understand the question. What do you want a JIRA for?

Does this fix the windows verification issue as well?

@pitrou

pitrou commented Nov 3, 2021

Copy link
Copy Markdown
Member Author

Does this fix the windows verification issue as well?

Hopefully, but I have no way to verify it (as explained above :-)).

@pitrou

pitrou commented Nov 3, 2021

Copy link
Copy Markdown
Member Author

(see diff in https://github.com/apache/arrow/pull/11583/files#diff-6c1e203cf55c63a1f177b4b0c97dc1aa62ea5eed5bd8e017e70703ddf119e343L90-L92)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants