Skip to content

[SPARK-42929][CONNECT] make mapInPandas / mapInArrow support "is_barrier"#40559

Closed
WeichenXu123 wants to merge 4 commits into
apache:masterfrom
WeichenXu123:spark-connect-barrier-mode
Closed

[SPARK-42929][CONNECT] make mapInPandas / mapInArrow support "is_barrier"#40559
WeichenXu123 wants to merge 4 commits into
apache:masterfrom
WeichenXu123:spark-connect-barrier-mode

Conversation

@WeichenXu123

@WeichenXu123 WeichenXu123 commented Mar 27, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

make mapInPandas / mapInArrow support "is_barrier"

Why are the changes needed?

feature parity.

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

Manually:

bin/pyspark --remote local:

from pyspark.sql.functions import pandas_udf
df = spark.createDataFrame([(1, 21), (2, 30)], ("id", "age"))
def filter_func(iterator):
    for pdf in iterator:
        yield pdf[pdf.id == 1]
df.mapInPandas(filter_func, df.schema,  is_barrier=True).collect()

def filter_func(iterator):
    for batch in iterator:
        pdf = batch.to_pandas()
        yield pyarrow.RecordBatch.from_pandas(pdf[pdf.id == 1])

df.mapInArrow(filter_func, df.schema, is_barrier=True).collect()

@WeichenXu123
init
662c88e 
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
@WeichenXu123 WeichenXu123 marked this pull request as draft March 27, 2023 07:03
@WeichenXu123
update
0703bb0 
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
zhengruifeng
zhengruifeng approved these changes Mar 27, 2023
View reviewed changes

@zhengruifeng zhengruifeng left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhengruifeng

zhengruifeng commented Mar 27, 2023

Copy link
Copy Markdown
Contributor

I guess you forgot to run dev/connect-gen-protos.sh ?

@WeichenXu123
gen proto
d8a175d 
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
@WeichenXu123 WeichenXu123 marked this pull request as ready for review March 27, 2023 08:01
@WeichenXu123
format
f8017da 
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
@WeichenXu123

WeichenXu123 commented Mar 27, 2023

Copy link
Copy Markdown
Contributor Author

Merged to master

@WeichenXu123 WeichenXu123 changed the title [SPARK-42929] make mapInPandas / mapInArrow support "is_barrier" [SPARK-42929][CONNECT] make mapInPandas / mapInArrow support "is_barrier" Mar 27, 2023
@ueshin ueshin mentioned this pull request Mar 28, 2023
Closed
HyukjinKwon pushed a commit that referenced this pull request Mar 29, 2023
@ueshin @HyukjinKwon
[SPARK-42929][CONNECT][FOLLOWUP] Rename isBarrier to barrier
aacac46 
### What changes were proposed in this pull request?

This is a follow-up of #40559 and #40571.

Renames `isBarrier` to `barrier` in Spark Connect, too.

### Why are the changes needed?

#40571 changed the argument name from `isBarrier` to `barrier`, so Spark Connect should follow it.

### Does this PR introduce _any_ user-facing change?

Yes, it renames the parameter.

### How was this patch tested?

Existing tests.

Closes #40579 from ueshin/issues/SPARK-42929/barrier.

Authored-by: Takuya UESHIN <ueshin@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
@wbo4958 wbo4958 mentioned this pull request Jan 26, 2024
Closed
@wbo4958 wbo4958 mentioned this pull request Feb 28, 2024
Closed
HyukjinKwon pushed a commit that referenced this pull request Feb 28, 2024
@wbo4958 @HyukjinKwon
[SPARK-42929][CONNECT][PYTHON][TEST] test barrier mode for mapInPanda…
34026d1 
…s/mapInArrow

### What changes were proposed in this pull request?

Add barrier mode tests for mapInPandas and mapInArrow.

### Why are the changes needed?

This is the follow-up of #40559

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

The newly added tests can pass the CIs

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #45310 from wbo4958/barrier-test.

Authored-by: Bobby Wang <wbo4958@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
TakawaAkirayo pushed a commit to TakawaAkirayo/spark that referenced this pull request Mar 4, 2024
@wbo4958 @TakawaAkirayo
[SPARK-42929][CONNECT][PYTHON][TEST] test barrier mode for mapInPanda…
129f603 
…s/mapInArrow

### What changes were proposed in this pull request?

Add barrier mode tests for mapInPandas and mapInArrow.

### Why are the changes needed?

This is the follow-up of apache#40559

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

The newly added tests can pass the CIs

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#45310 from wbo4958/barrier-test.

Authored-by: Bobby Wang <wbo4958@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
ericm-db pushed a commit to ericm-db/spark that referenced this pull request Mar 5, 2024
@wbo4958 @ericm-db
[SPARK-42929][CONNECT][PYTHON][TEST] test barrier mode for mapInPanda…
ba69cdf 
…s/mapInArrow

### What changes were proposed in this pull request?

Add barrier mode tests for mapInPandas and mapInArrow.

### Why are the changes needed?

This is the follow-up of apache#40559

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

The newly added tests can pass the CIs

### Was this patch authored or co-authored using generative AI tooling?
No

Closes apache#45310 from wbo4958/barrier-test.

Authored-by: Bobby Wang <wbo4958@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes
@zhengruifeng zhengruifeng zhengruifeng approved these changes

Assignees

No one assigned

Labels

CONNECT CORE PYTHON SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@WeichenXu123 @zhengruifeng @HyukjinKwon