[SPARK-31102][SQL][3.0] Spark-sql fails to parse when contains comment by javierivanov · Pull Request #28565 · apache/spark

javierivanov · 2020-05-18T03:51:18Z

This PR introduces a change to false for the insideComment flag on a newline. Fixing the issue introduced by SPARK-30049.

Backport to 3.0 from #27920

Previously on SPARK-30049 a comment containing an unclosed quote produced the following issue:

spark-sql> SELECT 1 -- someone's comment here
         > ;
Error in query:
extraneous input ';' expecting <EOF>(line 2, pos 0)

== SQL ==
SELECT 1 -- someone's comment here
;
^^^

This was caused because there was no flag for comment sections inside the splitSemiColon method to ignore quotes. SPARK-30049 added that flag and fixed the issue, but introduced the follwoing problem:

spark-sql> select
         >   1,
         >   -- two
         >   2;
Error in query:
mismatched input '<EOF>' expecting {'(', 'ADD', 'AFTER', 'ALL', 'ALTER', ...}(line 3, pos 2)
== SQL ==
select
  1,
--^^^

This issue is generated by a missing turn-off for the insideComment flag with a newline.

No

For previous tests using line-continuity(\) it was added a line-continuity rule in the SqlBase.g4 file to add the functionality to the SQL context.
A new test for inline comments was added.

Closes #27920 from javierivanov/SPARK-31102.

Authored-by: Javier Fuentes j.fuentes.m@icloud.com
Signed-off-by: Wenchen Fan wenchen@databricks.com

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

This PR introduces a change to false for the insideComment flag on a newline. Fixing the issue introduced by SPARK-30049. Previously on SPARK-30049 a comment containing an unclosed quote produced the following issue: ``` spark-sql> SELECT 1 -- someone's comment here > ; Error in query: extraneous input ';' expecting <EOF>(line 2, pos 0) == SQL == SELECT 1 -- someone's comment here ; ^^^ ``` This was caused because there was no flag for comment sections inside the splitSemiColon method to ignore quotes. SPARK-30049 added that flag and fixed the issue, but introduced the follwoing problem: ``` spark-sql> select > 1, > -- two > 2; Error in query: mismatched input '<EOF>' expecting {'(', 'ADD', 'AFTER', 'ALL', 'ALTER', ...}(line 3, pos 2) == SQL == select 1, --^^^ ``` This issue is generated by a missing turn-off for the insideComment flag with a newline. No - For previous tests using line-continuity(`\`) it was added a line-continuity rule in the SqlBase.g4 file to add the functionality to the SQL context. - A new test for inline comments was added. Closes apache#27920 from javierivanov/SPARK-31102. Authored-by: Javier Fuentes <j.fuentes.m@icloud.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

cloud-fan · 2020-05-18T04:32:53Z

ok to test

SparkQA · 2020-05-18T04:47:16Z

Test build #122783 has finished for PR 28565 at commit 49fa2a8.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-05-18T05:34:33Z

retest this please

SparkQA · 2020-05-18T05:48:43Z

Test build #122788 has finished for PR 28565 at commit 49fa2a8.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-05-18T05:55:48Z

It seems the failure above is not related to this PR. See: #28566

HyukjinKwon · 2020-05-18T05:58:13Z

retest this please

maropu · 2020-05-18T06:04:00Z

Looks nice! Thanks for re-trigger the tests, @HyukjinKwon .

SparkQA · 2020-05-18T07:05:03Z

Test build #122789 has finished for PR 28565 at commit 49fa2a8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-05-18T07:45:23Z

retest this please

SparkQA · 2020-05-18T10:14:59Z

Test build #122795 has finished for PR 28565 at commit 49fa2a8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-05-18T10:47:06Z

The newly added test all fails. Seems there is something different in branch 3.0.

javierivanov · 2020-05-18T11:41:24Z

The newly added test all fails. Seems there is something different in branch 3.0.

I added those test by mistake while cherry-picking. Checking.

SparkQA · 2020-05-18T18:50:42Z

Test build #122805 has finished for PR 28565 at commit a318127.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

This PR introduces a change to false for the insideComment flag on a newline. Fixing the issue introduced by SPARK-30049. Backport to 3.0 from #27920 Previously on SPARK-30049 a comment containing an unclosed quote produced the following issue: ``` spark-sql> SELECT 1 -- someone's comment here > ; Error in query: extraneous input ';' expecting <EOF>(line 2, pos 0) == SQL == SELECT 1 -- someone's comment here ; ^^^ ``` This was caused because there was no flag for comment sections inside the splitSemiColon method to ignore quotes. SPARK-30049 added that flag and fixed the issue, but introduced the follwoing problem: ``` spark-sql> select > 1, > -- two > 2; Error in query: mismatched input '<EOF>' expecting {'(', 'ADD', 'AFTER', 'ALL', 'ALTER', ...}(line 3, pos 2) == SQL == select 1, --^^^ ``` This issue is generated by a missing turn-off for the insideComment flag with a newline. No - For previous tests using line-continuity(`\`) it was added a line-continuity rule in the SqlBase.g4 file to add the functionality to the SQL context. - A new test for inline comments was added. Closes #27920 from javierivanov/SPARK-31102. Authored-by: Javier Fuentes <j.fuentes.micloud.com> Signed-off-by: Wenchen Fan <wenchendatabricks.com> ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Closes #28565 from javierivanov/SPARK-3.0-31102. Authored-by: Javier Fuentes <j.fuentes.m@icloud.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>

maropu · 2020-05-18T23:57:56Z

Thanks! Merged to branch-3.0.

probot-autolabeler Bot added the SQL label May 18, 2020

cloud-fan approved these changes May 18, 2020

View reviewed changes

maropu approved these changes May 18, 2020

View reviewed changes

removed tests carried from master

a318127

maropu closed this May 18, 2020

Uh oh!

Conversation

javierivanov commented May 18, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

cloud-fan commented May 18, 2020

Uh oh!

SparkQA commented May 18, 2020

Uh oh!

maropu commented May 18, 2020

Uh oh!

SparkQA commented May 18, 2020

Uh oh!

maropu commented May 18, 2020

Uh oh!

HyukjinKwon commented May 18, 2020

Uh oh!

maropu commented May 18, 2020

Uh oh!

SparkQA commented May 18, 2020

Uh oh!

AngersZhuuuu commented May 18, 2020

Uh oh!

SparkQA commented May 18, 2020

Uh oh!

cloud-fan commented May 18, 2020

Uh oh!

javierivanov commented May 18, 2020

Uh oh!

SparkQA commented May 18, 2020

Uh oh!

maropu commented May 18, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants