[SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7 by cloud-fan · Pull Request #20584 · apache/spark

cloud-fan · 2018-02-12T04:29:54Z

What changes were proposed in this pull request?

This test only fails with sbt on Hadoop 2.7, I can't reproduce it locally, but here is my speculation by looking at the code:

FileSystem.delete doesn't delete the directory entirely, somehow we can still open the file as a 0-length empty file.(just speculation)
ORC intentionally allow empty files, and the reader fails during reading without closing the file stream.

This PR improves the test to make sure all files are deleted and can't be opened.

How was this patch tested?

N/A

cloud-fan · 2018-02-12T04:30:42Z

cc @sameeragarwal @dongjoon-hyun @gatorsmile

HyukjinKwon · 2018-02-12T05:05:04Z

-          val fs = thirdPath.getFileSystem(spark.sparkContext.hadoopConfiguration)
+          // Make sure all data files are deleted and can't be opened.
+          files.foreach(f => fs.delete(f, false))
          assert(fs.delete(thirdPath, true))


Hmmmm .. but it asserts true on the delete completion .. I would be surprised if it's something not guaranteed ..

HyukjinKwon · 2018-02-12T05:15:22Z

BTW, my rough wild guess was that case 2. (reading it but not closing it) happens in schema inference path.

gatorsmile · 2018-02-12T05:19:45Z

LGTM, I would merge this first if Jenkins does not fail and see whether this can help fix the flaky tests.

HyukjinKwon · 2018-02-12T05:51:20Z

I won't get in the way but I am less sure on this. I thought this is also flaky in PR builder too anyway.

cloud-fan · 2018-02-12T05:54:15Z

BTW, my rough wild guess was that case 2. (reading it but not closing it) happens in schema inference path.

According to the log, the leaked file stream was created when building the ORC columnar reader. Schema inference is fine.

HyukjinKwon · 2018-02-12T05:58:56Z

You are right. I have run out of ideas. LGTM too for a try if it happens more frequently in spark-branch-2.3-test-sbt-hadoop-2.7.

sameeragarwal · 2018-02-12T06:06:46Z

LGTM, seems plausible!

kiszk · 2018-02-12T07:07:12Z

I am also thinking about this. I agree with this.

According to the log, the leaked file stream was created when building the ORC columnar reader.

I am suspicious about relationship between afterEach() and addTaskCompletionListener (call close()). But, not sure. Let us try this approach first.

SparkQA · 2018-02-12T07:31:52Z

Test build #87321 has finished for PR 20584 at commit 51bb48a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sameeragarwal · 2018-02-12T07:37:36Z

merging this to master/2.3. Thanks!

…k 2.3/hadoop 2.7 ## What changes were proposed in this pull request? This test only fails with sbt on Hadoop 2.7, I can't reproduce it locally, but here is my speculation by looking at the code: 1. FileSystem.delete doesn't delete the directory entirely, somehow we can still open the file as a 0-length empty file.(just speculation) 2. ORC intentionally allow empty files, and the reader fails during reading without closing the file stream. This PR improves the test to make sure all files are deleted and can't be opened. ## How was this patch tested? N/A Author: Wenchen Fan <wenchen@databricks.com> Closes #20584 from cloud-fan/flaky-test. (cherry picked from commit 6efd5d1) Signed-off-by: Sameer Agarwal <sameerag@apache.org>

cloud-fan · 2018-02-12T13:30:38Z

I am suspicious about relationship between afterEach() and addTaskCompletionListener (call close()). But, not sure. Let us try this approach first.

This is one of my speculations. There 2 possibilities I can think of: 1) the task completion listener is not called before afterEach. 2) the orc columnar reader's close method doesn't close the file stream.

For 1), seems we've fixed it in c5a31d1 . For 2), I'm not sure and may need help from ORC folks.

HyukjinKwon · 2018-02-12T13:32:51Z

I think I rushed to take a look Initially. Thanks for fixing this.

…k 2.3/hadoop 2.7 ## What changes were proposed in this pull request? This test only fails with sbt on Hadoop 2.7, I can't reproduce it locally, but here is my speculation by looking at the code: 1. FileSystem.delete doesn't delete the directory entirely, somehow we can still open the file as a 0-length empty file.(just speculation) 2. ORC intentionally allow empty files, and the reader fails during reading without closing the file stream. This PR improves the test to make sure all files are deleted and can't be opened. ## How was this patch tested? N/A Author: Wenchen Fan <wenchen@databricks.com> Closes apache#20584 from cloud-fan/flaky-test.

gatorsmile · 2018-02-12T16:38:51Z

Great! https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/ becomes green again!!!

dongjoon-hyun · 2018-02-12T17:15:06Z

My bad. Thank you, guys. For the following, I'll investigate it more.

According to the log, the leaked file stream was created when building the ORC columnar reader.

dongjoon-hyun · 2018-02-12T20:05:18Z

This patch helps branch-2.3/sbt/hadoop2.7. So, I'm seriously monitoring the latest consecutive failures at sbt and hadoop-2.6 at master branch, too.

4210 Passed!
4209 Failed with FileBasedDataSourceSuite and ParquetQuerySuite
4208 This patch landed here but failed with StreamingOuterJoinSuite and ReceiverSuite.
4207 Failed with ParquetQuerySuite
4206 Failed with BufferHolderSparkSubmitSuite

dongjoon-hyun · 2018-02-12T20:27:26Z

For the following case, I'll make a PR for Spark ORC columnar reader very soon.

the orc columnar reader's close method doesn't close the file stream.

dongjoon-hyun · 2018-02-12T22:05:33Z

I created a PR, #20590 .

make sure all files are deleted when testing IGNORE_MISSING_FILES

51bb48a

HyukjinKwon reviewed Feb 12, 2018

View reviewed changes

asfgit closed this in 6efd5d1 Feb 12, 2018

Uh oh!

Conversation

cloud-fan commented Feb 12, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

cloud-fan commented Feb 12, 2018

Uh oh!

HyukjinKwon Feb 12, 2018

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Feb 12, 2018

Uh oh!

gatorsmile commented Feb 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Feb 12, 2018

Uh oh!

cloud-fan commented Feb 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Feb 12, 2018

Uh oh!

sameeragarwal commented Feb 12, 2018

Uh oh!

kiszk commented Feb 12, 2018

Uh oh!

SparkQA commented Feb 12, 2018

Uh oh!

sameeragarwal commented Feb 12, 2018

Uh oh!

cloud-fan commented Feb 12, 2018

Uh oh!

HyukjinKwon commented Feb 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Feb 12, 2018

Uh oh!

dongjoon-hyun commented Feb 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Feb 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Feb 12, 2018

Uh oh!

dongjoon-hyun commented Feb 12, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

gatorsmile commented Feb 12, 2018 •

edited

Loading

cloud-fan commented Feb 12, 2018 •

edited

Loading

HyukjinKwon commented Feb 12, 2018 •

edited

Loading

dongjoon-hyun commented Feb 12, 2018 •

edited

Loading

dongjoon-hyun commented Feb 12, 2018 •

edited

Loading