Skip to content

[SPARK-34837][SQL][FOLLOWUP] Fix division by zero in the avg function over ANSI intervals#32358

Closed
beliefer wants to merge 1 commit into
apache:masterfrom
beliefer:SPARK-34837-followup
Closed

[SPARK-34837][SQL][FOLLOWUP] Fix division by zero in the avg function over ANSI intervals#32358
beliefer wants to merge 1 commit into
apache:masterfrom
beliefer:SPARK-34837-followup

Conversation

@beliefer

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

#32229 support ANSI SQL intervals by the aggregate function avg.
But have not treat that the input zero rows. so this will lead to:

Caused by: java.lang.ArithmeticException: / by zero
	at com.google.common.math.LongMath.divide(LongMath.java:367)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1864)
	at org.apache.spark.rdd.RDD.$anonfun$count$1(RDD.scala:1253)
	at org.apache.spark.rdd.RDD.$anonfun$count$1$adapted(RDD.scala:1253)
	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2248)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1437)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Why are the changes needed?

Fix a bug.

Does this PR introduce any user-facing change?

No. Just new feature.

How was this patch tested?

new tests.

@github-actions github-actions Bot added the SQL label Apr 27, 2021

@MaxGekk MaxGekk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about to make PR title more precise:
[SPARK-34837][SQL][FOLLOWUP] Fix division by zero in the avg function over ANSI intervals

@beliefer beliefer changed the title [SPARK-34837][SQL][FOLLOWUP] Support ANSI SQL intervals by the aggregate function avg [SPARK-34837][SQL][FOLLOWUP] Fix division by zero in the avg function over ANSI intervals Apr 27, 2021
@SparkQA

SparkQA commented Apr 27, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42503/

@SparkQA

SparkQA commented Apr 27, 2021

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42503/

@MaxGekk

MaxGekk commented Apr 27, 2021

Copy link
Copy Markdown
Member

+1, LGTM. Merging to master.
Thank you, @beliefer .

@MaxGekk MaxGekk closed this in 55dea2d Apr 27, 2021
@SparkQA

SparkQA commented Apr 27, 2021

Copy link
Copy Markdown

Test build #137983 has finished for PR 32358 at commit 02d8282.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@beliefer

Copy link
Copy Markdown
Contributor Author

@MaxGekk Thanks for you review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants