Skip to content

[SPARK-19599][SS]Clean up HDFSMetadataLog#16932

Closed
zsxwing wants to merge 3 commits into
apache:masterfrom
zsxwing:metadata-cleanup
Closed

[SPARK-19599][SS]Clean up HDFSMetadataLog#16932
zsxwing wants to merge 3 commits into
apache:masterfrom
zsxwing:metadata-cleanup

Conversation

@zsxwing

@zsxwing zsxwing commented Feb 14, 2017

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

SPARK-19464 removed support for Hadoop 2.5 and earlier, so we can do some cleanup for HDFSMetadataLog.

This PR includes the following changes:

  • Remove the workaround codes for HADOOP-10622. Unfortunately, there is another issue HADOOP-14084 that prevents us from removing the workaround codes.
  • Remove unnecessary writer: (T, OutputStream) => Unit and just call serialize directly.
  • Remove catching FileNotFoundException.

How was this patch tested?

Jenkins

// So throw an exception to tell the user this is not a valid behavior.
throw new ConcurrentModificationException(
s"Multiple HDFSMetadataLog are using $path", e)
case e: FileNotFoundException =>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed it because we always create a temp file so it won't happen.

@SparkQA

SparkQA commented Feb 14, 2017

Copy link
Copy Markdown

Test build #72890 has finished for PR 16932 at commit 81bcc48.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing

zsxwing commented Feb 14, 2017

Copy link
Copy Markdown
Member Author

retest this please

@SparkQA

SparkQA commented Feb 15, 2017

Copy link
Copy Markdown

Test build #72899 has finished for PR 16932 at commit 14a199c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing

zsxwing commented Feb 15, 2017

Copy link
Copy Markdown
Member Author

retest this please

@SparkQA

SparkQA commented Feb 15, 2017

Copy link
Copy Markdown

Test build #72898 has finished for PR 16932 at commit 81bcc48.

  • This patch fails from timeout after a configured wait of `250m`.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Feb 15, 2017

Copy link
Copy Markdown

Test build #72906 has finished for PR 16932 at commit 14a199c.

  • This patch fails from timeout after a configured wait of `250m`.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing zsxwing changed the title [SPARK-19599][SS]Clean up HDFSMetadataLog for Hadoop 2.6+ [SPARK-19599][SS]Clean up HDFSMetadataLog Feb 15, 2017
@brkyvz

brkyvz commented Feb 15, 2017

Copy link
Copy Markdown
Contributor

LGTM!

@SparkQA

SparkQA commented Feb 15, 2017

Copy link
Copy Markdown

Test build #72954 has finished for PR 16932 at commit 38444ea.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing

zsxwing commented Feb 16, 2017

Copy link
Copy Markdown
Member Author

Thanks! Merging to master.

asfgit pushed a commit that referenced this pull request Feb 16, 2017
## What changes were proposed in this pull request?

SPARK-19464 removed support for Hadoop 2.5 and earlier, so we can do some cleanup for HDFSMetadataLog.

This PR includes the following changes:
- ~~Remove the workaround codes for HADOOP-10622.~~ Unfortunately, there is another issue [HADOOP-14084](https://issues.apache.org/jira/browse/HADOOP-14084) that prevents us from removing the workaround codes.
- Remove unnecessary `writer: (T, OutputStream) => Unit` and just call `serialize` directly.
- Remove catching FileNotFoundException.

## How was this patch tested?

Jenkins

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #16932 from zsxwing/metadata-cleanup.

(cherry picked from commit 21b4ba2)
Signed-off-by: Shixiong Zhu <shixiong@databricks.com>
@zsxwing

zsxwing commented Feb 16, 2017

Copy link
Copy Markdown
Member Author

Since this change is minor and safe, I also cherry-picked to 2.1 to avoid merge conflicts in future.

@asfgit asfgit closed this in 21b4ba2 Feb 16, 2017
@srowen

srowen commented Feb 16, 2017

Copy link
Copy Markdown
Member

The final code change doesn't assume Hadoop 2.6 right?

@zsxwing zsxwing deleted the metadata-cleanup branch February 16, 2017 00:26
@zsxwing

zsxwing commented Feb 16, 2017

Copy link
Copy Markdown
Member Author

The final code change doesn't assume Hadoop 2.6 right?

Right.

cmonkey pushed a commit to cmonkey/spark that referenced this pull request Feb 16, 2017
## What changes were proposed in this pull request?

SPARK-19464 removed support for Hadoop 2.5 and earlier, so we can do some cleanup for HDFSMetadataLog.

This PR includes the following changes:
- ~~Remove the workaround codes for HADOOP-10622.~~ Unfortunately, there is another issue [HADOOP-14084](https://issues.apache.org/jira/browse/HADOOP-14084) that prevents us from removing the workaround codes.
- Remove unnecessary `writer: (T, OutputStream) => Unit` and just call `serialize` directly.
- Remove catching FileNotFoundException.

## How was this patch tested?

Jenkins

Author: Shixiong Zhu <shixiong@databricks.com>

Closes apache#16932 from zsxwing/metadata-cleanup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants