Skip to content

try to fix forward tests for java 17 #39058

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
derrickaw merged 5 commits into from
Jun 22, 2026
Merged

try to fix forward tests for java 17 #39058

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b39db41
try to fix forward with 17 only
derrickaw Jun 22, 2026
a3fea20
address just two workflows
derrickaw Jun 22, 2026
83f3984
Revert XVR Spark3 workflow java-version override
derrickaw Jun 22, 2026
c5f170d
Add Java 17 and 11 setup and -Pjava17Home across all XVR workflows
derrickaw Jun 22, 2026
4ae65b1
Revert redundant java-version override in TransformService Direct wor…
derrickaw Jun 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/beam_PostCommit_XVR_Flink.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,9 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
java-version: |
17
11
python-version: ${{ matrix.python_version }}
- name: run PostCommit XVR Flink script
env:
Expand All @@ -83,6 +86,7 @@ jobs:
with:
gradle-command: :runners:flink:${{ env.FlinkVersion }}:job-server:validatesCrossLanguageRunner
arguments: |
-Pjava17Home=$JAVA_HOME_17_X64 \
-PpythonVersion=${{ matrix.python_version }} \
-PskipNonPythonTask=${{ (matrix.python_version == '3.10' && true) || false }} \
- name: Archive JUnit Test Results
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/beam_PostCommit_XVR_GoUsingJava_Dataflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,9 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
java-version: |
17
11
python-version: default
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5
Expand All @@ -86,6 +89,7 @@ jobs:
with:
gradle-command: :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerGoUsingJava
arguments: |
-Pjava17Home=$JAVA_HOME_17_X64 \
-PuseDockerBuildx
- name: Archive JUnit Test Results
uses: actions/upload-artifact@v7
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/beam_PostCommit_XVR_JavaUsingPython_Dataflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
java-version: |
17
11
python-version: |
3.10
${{ matrix.python_version }}
Expand All @@ -82,6 +85,7 @@ jobs:
with:
gradle-command: :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerJavaUsingPython
arguments: |
-Pjava17Home=$JAVA_HOME_17_X64 \
-PpythonVersion=${{ matrix.python_version }} \
- name: Archive JUnit Test Results
uses: actions/upload-artifact@v7
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,12 +73,16 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
java-version: |
17
11
python-version: 3.14
- name: run PostCommit XVR PythonUsingJavaSQL Dataflow script
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingSql
arguments: |
-Pjava17Home=$JAVA_HOME_17_X64 \
-PpythonVersion=3.14 \
- name: Archive Python Test Results
uses: actions/upload-artifact@v7
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/beam_PostCommit_XVR_PythonUsingJava_Dataflow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
java-version: |
17
11
python-version: |
3.10
${{ matrix.python_version }}
Expand All @@ -82,6 +85,7 @@ jobs:
with:
gradle-command: :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava
arguments: |
-Pjava17Home=$JAVA_HOME_17_X64 \
-PpythonVersion=${{ matrix.python_version }} \
- name: Archive Python Test Results
uses: actions/upload-artifact@v7
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/beam_PostCommit_XVR_Spark3.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
java-version: |
17
11
python-version: ${{ matrix.python_version }}
- name: run PostCommit XVR Spark3 script
env:
Expand All @@ -82,6 +85,7 @@ jobs:
with:
gradle-command: :runners:spark:3:job-server:validatesCrossLanguageRunner
arguments: |
-Pjava17Home=$JAVA_HOME_17_X64 \
-PpythonVersion=${{ matrix.python_version }} \
-PskipNonPythonTask=${{ (matrix.python_version == '3.10' && true) || false }} \
- name: Archive JUnit Test Results
Expand Down
24 changes: 15 additions & 9 deletions buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -978,6 +978,9 @@ class BeamModulePlugin implements Plugin<Project> {

// set compiler options for java version overrides to compile with a different java version
project.ext.setJavaVerOptions = { CompileOptions options, String ver ->
if (!project.findProperty("java${ver}Home")) {
return
}
if (ver == '8') {
def java8Home = project.findProperty("java8Home")
options.fork = true
Expand Down Expand Up @@ -1161,16 +1164,16 @@ class BeamModulePlugin implements Plugin<Project> {
// If compiled on older SDK, compile with JDK configured with compatible javaXXHome
// The order is intended here
if (requireJavaVersion.compareTo(JavaVersion.VERSION_11) <= 0 &&
project.hasProperty('java11Home')) {
project.findProperty('java11Home')) {
forkJavaVersion = '11'
} else if (requireJavaVersion.compareTo(JavaVersion.VERSION_17) <= 0 &&
project.hasProperty('java17Home')) {
project.findProperty('java17Home')) {
forkJavaVersion = '17'
} else if (requireJavaVersion.compareTo(JavaVersion.VERSION_21) <= 0 &&
project.hasProperty('java21Home')) {
project.findProperty('java21Home')) {
forkJavaVersion = '21'
} else if (requireJavaVersion.compareTo(JavaVersion.VERSION_25) <= 0 &&
project.hasProperty('java25Home')) {
project.findProperty('java25Home')) {
forkJavaVersion = '25'
} else {
logger.config("Module ${project.name} disabled. To enable, either " +
Expand Down Expand Up @@ -1628,7 +1631,7 @@ class BeamModulePlugin implements Plugin<Project> {
}
// if specified test java version, modify the compile and runtime versions accordingly
if (['11', '17', '21', '25'].contains(testJavaVersion)) {
def testJavaHome = project.getProperty("java${testJavaVersion}Home")
def testJavaHome = project.findProperty("java${testJavaVersion}Home")
Comment on lines 1633 to +1634

@gemini-code-assist gemini-code-assist Bot Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If testJavaVersion is specified but the corresponding java${testJavaVersion}Home property is not defined, project.findProperty will return null. This can lead to a NullPointerException or IllegalArgumentException when configuring the compilation and test tasks later in this block.\n\nWe can make this safer by checking if the property is defined directly in the if condition.

      if (['11', '17', '21', '25'].contains(testJavaVersion) && project.findProperty("java${testJavaVersion}Home")) {\n        def testJavaHome = project.findProperty("java${testJavaVersion}Home")


// redirect java compiler to specified version for compileTestJava only
project.tasks.compileTestJava {
Expand Down Expand Up @@ -2710,7 +2713,8 @@ class BeamModulePlugin implements Plugin<Project> {

def pythonDir = project.project(":sdks:python").projectDir
def usesDataflowRunner = config.pythonPipelineOptions.contains("--runner=TestDataflowRunner") || config.pythonPipelineOptions.contains("--runner=DataflowRunner")
def javaContainerSuffix = getSupportedJavaVersion()
String ver = project.findProperty('testJavaVersion')
def javaContainerSuffix = ver ? getSupportedJavaVersion(ver) : getSupportedJavaVersion()

// Sets up, collects, and runs Python pipeline tests
project.tasks.register(config.name+"PythonUsingJava") {
Expand Down Expand Up @@ -2790,7 +2794,8 @@ class BeamModulePlugin implements Plugin<Project> {
]
def serviceArgs = project.project(':sdks:python').mapToArgString(expansionServiceOpts)
def pythonContainerSuffix = project.project(':sdks:python').pythonVersion.replace('.', '')
def javaContainerSuffix = getSupportedJavaVersion()
String ver = project.findProperty('testJavaVersion')
def javaContainerSuffix = ver ? getSupportedJavaVersion(ver) : getSupportedJavaVersion()
def setupTask = project.tasks.register(config.name+"Setup", Exec) {
dependsOn ':sdks:java:container:'+javaContainerSuffix+':docker'
dependsOn ':sdks:python:container:py'+pythonContainerSuffix+':docker'
Expand Down Expand Up @@ -2979,10 +2984,11 @@ class BeamModulePlugin implements Plugin<Project> {
]
def serviceArgs = project.project(':sdks:python').mapToArgString(transformServiceOpts)
def pythonContainerSuffix = project.project(':sdks:python').pythonVersion.replace('.', '')
def javaContainerSuffix = getSupportedJavaVersion()
String ver = project.findProperty('testJavaVersion')
def javaContainerSuffix = ver ? getSupportedJavaVersion(ver) : getSupportedJavaVersion()

// Transform service delivers transforms that refer to SDK harness containers with following sufixes.
def transformServiceJavaContainerSuffix = 'java11'
def transformServiceJavaContainerSuffix = 'java17'

@gemini-code-assist gemini-code-assist Bot Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Hardcoding transformServiceJavaContainerSuffix to 'java17' will cause test failures when running tests on other Java versions (such as Java 11 or Java 21). This is because the setup task only builds the container image specified by javaContainerSuffix (which is dynamic based on testJavaVersion). If the transform service tries to use 'java17' but only 'java11' was built, the test will fail with an image not found error.\n\nWe should make transformServiceJavaContainerSuffix dynamic by setting it to javaContainerSuffix so that it always matches the container version being built and tested.

      def transformServiceJavaContainerSuffix = javaContainerSuffix

def transformServicePythonContainerSuffix = pythonContainerSuffix

def setupTask = project.tasks.register(config.name+"Setup", Exec) {
Expand Down
2 changes: 1 addition & 1 deletion sdks/java/transform-service/controller-container/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
# limitations under the License.
###############################################################################

FROM eclipse-temurin:11
FROM eclipse-temurin:17
LABEL Author "Apache Beam <dev@beam.apache.org>"
ARG TARGETOS
ARG TARGETARCH
Expand Down
Loading