Skip to content

[SPARK-18368] [SQL] Fix regexp replace when serialized#15834

Closed
rdblue wants to merge 2 commits into
apache:masterfrom
rdblue:SPARK-18368-fix-regexp-replace
Closed

[SPARK-18368] [SQL] Fix regexp replace when serialized#15834
rdblue wants to merge 2 commits into
apache:masterfrom
rdblue:SPARK-18368-fix-regexp-replace

Conversation

@rdblue

@rdblue rdblue commented Nov 9, 2016

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This makes the result value both transient and lazy, so that if the RegExpReplace object is initialized then serialized, result: StringBuffer will be correctly initialized.

How was this patch tested?

  • Verified that this patch fixed the query that found the bug.
  • Added a test case that fails without the fix.

@rdblue

rdblue commented Nov 9, 2016

Copy link
Copy Markdown
Contributor Author

@yhuai, this replaces #15816. The ref, 3536f6a, has already passed tests on that PR so it should be safe to merge this.

@yhuai

yhuai commented Nov 9, 2016

Copy link
Copy Markdown
Contributor

Awesome! btw looks like your original changes in ExpressionEvalHelper.scala (https://github.com/apache/spark/pull/15816/files#diff-41747ec3f56901eb7bfb95d2a217e94d) uncovered issues with other expressions. Want to send a new pr to fix them as well?

@yhuai

yhuai commented Nov 9, 2016

Copy link
Copy Markdown
Contributor

image

Since tests have already passed, I am merging this PR to master/branch-2.1/branch-2.0.

@rdblue

rdblue commented Nov 9, 2016

Copy link
Copy Markdown
Contributor Author

SPARK-18387 tracks the other bugs.

@yhuai

yhuai commented Nov 9, 2016

Copy link
Copy Markdown
Contributor

Great. Thanks!

asfgit pushed a commit that referenced this pull request Nov 9, 2016
## What changes were proposed in this pull request?

This makes the result value both transient and lazy, so that if the RegExpReplace object is initialized then serialized, `result: StringBuffer` will be correctly initialized.

## How was this patch tested?

* Verified that this patch fixed the query that found the bug.
* Added a test case that fails without the fix.

Author: Ryan Blue <blue@apache.org>

Closes #15834 from rdblue/SPARK-18368-fix-regexp-replace.

(cherry picked from commit d4028de)
Signed-off-by: Yin Huai <yhuai@databricks.com>
asfgit pushed a commit that referenced this pull request Nov 9, 2016
## What changes were proposed in this pull request?

This makes the result value both transient and lazy, so that if the RegExpReplace object is initialized then serialized, `result: StringBuffer` will be correctly initialized.

## How was this patch tested?

* Verified that this patch fixed the query that found the bug.
* Added a test case that fails without the fix.

Author: Ryan Blue <blue@apache.org>

Closes #15834 from rdblue/SPARK-18368-fix-regexp-replace.

(cherry picked from commit d4028de)
Signed-off-by: Yin Huai <yhuai@databricks.com>
@asfgit asfgit closed this in d4028de Nov 9, 2016
@SparkQA

SparkQA commented Nov 9, 2016

Copy link
Copy Markdown

Test build #68416 has finished for PR 15834 at commit 3536f6a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
## What changes were proposed in this pull request?

This makes the result value both transient and lazy, so that if the RegExpReplace object is initialized then serialized, `result: StringBuffer` will be correctly initialized.

## How was this patch tested?

* Verified that this patch fixed the query that found the bug.
* Added a test case that fails without the fix.

Author: Ryan Blue <blue@apache.org>

Closes apache#15834 from rdblue/SPARK-18368-fix-regexp-replace.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants