Skip to content

Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)"#66952

Open
olegkachur-e wants to merge 1 commit into
apache:mainfrom
VladaZakharova:revert_beam_susp_manual
Open

Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)"#66952
olegkachur-e wants to merge 1 commit into
apache:mainfrom
VladaZakharova:revert_beam_susp_manual

Conversation

@olegkachur-e

@olegkachur-e olegkachur-e commented May 14, 2026

Copy link
Copy Markdown
Contributor

This reverts commit 917abea.

  • Remove hacks regarding the beam provider suspension, as they are not relevant anymore.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Important

🛠️ Maintainer triage note for @olegkachur-e · by @potiuk · 2026-07-02 17:46 UTC

Some review feedback from @eladkal is waiting on you:

  • 1 unresolved review thread(s) from @eladkal need a reply or a fix.

The ball is in your court — you've been assigned to this PR. Reply or push a fix in each thread, then mark them resolved.

Automated triage — may be imperfect; a maintainer takes the next look.

@potiuk

potiuk commented Jun 1, 2026

Copy link
Copy Markdown
Member

@olegkachur-e A few things need addressing before review — see our Pull Request quality criteria.

  • Unit tests. See docs.
  • Provider tests. See docs.
  • Other failing CI checks. See docs.
  • Image build. See docs.

No rush.


Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

@MaksYermak MaksYermak force-pushed the revert_beam_susp_manual branch 8 times, most recently from 1647831 to 794fc1a Compare June 23, 2026 08:42
@olegkachur-e olegkachur-e marked this pull request as ready for review June 23, 2026 11:40
Comment on lines +65 to +68
# For Python>=3.11, the apache-beam package depends on envoy-data-plane.
# Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these
# Python versions, and it explicitly requires a prerelease version of betterproto.
"betterproto==2.0.0b6; python_version >= '3.11'",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not a fan of this.
this is a beta version from 2023. This is not a case where in a few days/weeks things will change and a stable version would be released, is it?

I am not in the details of this so I will leave @potiuk the call here but from my point of view we should not deliever stable release with depenedency on beta packages.
Who gurentees that tommorow the author won't remove them from PyPi with a thought that there won't be any production impact on anyone?

@potiuk potiuk Jun 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big fan either.

But I think thee newer version of envoy-data-plane has been questioned by others for example here: apache/beam#37854

And I see that this is what latest beam has:

envoy-data-plane<2,>=1.0.3; python_version >= "3.11"

And the 2.1.0 version of envoy-data-plane uses already betterproto2 - which is a maintained replacement of betterproto.

So maybe we can ask apache-beam to upgrade to the envoy-data-plane>=2 ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I commented in the issue.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not a big fan of this idea either.

The reason why I decided to add it was in that this beta version exists from 2023 and I am not sure that it will be release in the future. One more point for me to added it was that I noticed that Apache Beam SDK use this beta version in there generated requirements for containers. And I had idea that if they already had hardcoded it in apache-beam requirements, from which our provider already depends, maybe it makes sense temporary hardcoded it in provider as well for fixing constrains problem and unsuspend provider.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't tell beam what to so from thier side but Airflow should not do this.

@potiuk

potiuk commented Jun 25, 2026

Copy link
Copy Markdown
Member

I commented in the issue.

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

uv.lock on main just moved via #69198 ("[main] Upgrade important CI environment"), commit 38b2869 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

…#61926)"

This reverts commit 917abea.

- Remove hacks regarding the beam provider suspension, as they are not
  relevant anymore.

ISSUE: apache#66551
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants