[SPARK-9801][Streaming]Check if file exists before deleting temporary files.

[viadea]

Spark streaming deletes the temp file and backup files without checking if they exist or not

[tdas]

Please set the PR title as the problem or fix this PR has. Something like "Check if file exists before deleting temporary files."

[tdas]

LGTM. Will merge when tests pass and after you update the title.

[tdas]

This is ok to test

[viadea]

Done changing the title.

[viadea]

Hi Team,

I forgot to add one check, I wanted to add the check for backup file also:
fs.delete(backupFile, true) // just in case it exists
Should I file a new PR or edit this PR?

[viadea]

Hi Team,

I just added another commit to this PR.

[SparkQA]

Test build #1424 has finished for PR 8082 at commit 087daf0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

I am not if this is necessary. These files that are being deleted came from a recent listing, so they are very very likely to exist.

[viadea]

I agreed and I just made the change. Can you start another build and test?

[viadea]

Actually from the error message in Spark-9801, the only 2 checks I wanted to add were:
tempFile
backupFile

So I am fine to ignore the check for "file". Also for performance consideration.

[SparkQA]

Test build #1425 has finished for PR 8082 at commit fd143f2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #40347 has finished for PR 8082 at commit 242d05f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SpecificMutableProjection extends $

[tdas]

All right, test on the latest commit has passed. I am merging this to master and branch 1.5. Thanks!

[tdas]

Actually, I merged it to 1.4 and 1.3 branches as well.