[breaking] Make pyspark-client as default and pyspark package optional

[raphaelauv]

current apache.spark provider need the package pyspark ( more than 400mb )

where I would like to only use the spark-client ( spark-connect ) to trigger a spark job ( that is 1.5mb )

    @task.pyspark(conn_id="spark_connect")
    def my_job(spark,sc):
        df = spark.range(100).filter("id % 2 = 0")
        print(df.count())

# or 

    def my_pyspark_job(spark):
        df = spark.range(100).filter("id % 2 = 0")
        print(df.count())

    PySparkOperator(
        python_callable=my_pyspark_job, conn_id="spark_connect", task_id="spark_pyspark_job"
    )
        

this is a breaking change but it will make things lighter by default , wdyt ? thanks

( btw spark-client is only available since spark 4.0 and need "grpcio >= 1.67.0" ( that conflict with apache-beam ) )

[potiuk]

Good idea - but we need to wait until apache-beam fixes grpcio limit - maybe find if there is an existing issue or you can open a new issue with them ?

[raphaelauv]

Yes we need apache-beam to support grpcio>= 1.67

you already created it -> apache/beam#34081

[potiuk]

you already created it -> apache/beam#34081

Heh... Almost a year ago.. Maybe a time to follow up ?

[potiuk]

I raised my comments in the discussion grpc/grpc#37710 (comment)

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[raphaelauv]

No stale -> #61926

[eladkal]

This needs entry in the top of the provider changelog explaining users the reasoning and what they should do if they want to keep the behavior as is

[potiuk]

this is a breaking change but it will make things lighter by default , wdyt ? thanks

Yeah. Just add entry to changelog :)

[potiuk]

I also added `"[breaking]" in the title of the PR - I think that might be one of the ways @eladkal ? how we mark breaking changes. We discussed it before. I think it's very nice way to add "[breaking]" to the title - then the RM can remove it when preparing release notes. But we can also detect it automatically.

I am also going to employ LLM -- optionally, following the experience with auto-triage - to prepare the release notes - so this might naturally work when we use LLM to do it.

[eladkal]

@raphaelauv can you please add the entry to the top of the change log? I will merge the PR after

[raphaelauv]

hey @eladkal , I rebased and added the comment on the "why"

Thanks