[ZEPPELIN-1515] Use Hadoop-FileSystem api to implement NotebookRepo#2529
[ZEPPELIN-1515] Use Hadoop-FileSystem api to implement NotebookRepo#2529prabhjyotsingh wants to merge 2 commits into
Conversation
prabhjyotsingh
changed the title
|
@zjffdu @jongyoul @felixcheung @halidhuseynov Please help review this. |
|
PING |
| </property> | ||
|
|
||
| <property> | ||
| <name>kerberos.principal</name> |
There was a problem hiding this comment.
should this be prefix with zeppelin.hadoop? there might be other uses of kerberos keytab/principal?
There was a problem hiding this comment.
like the key refresh stuff for shell interpreter...
| <name>zeppelin.hadoop.uri</name> | ||
| <value>hdfs://localhost:8020</value> | ||
| <description>The scheme of the URI determines a configuration property name, fs.scheme.class whose value names the FileSystem class. | ||
| e.g. `hdfs://localhost:8020`, or `file:///` |
There was a problem hiding this comment.
I guess this might be confused with webhdfs server url? I'm not sure if there is a better name for it than hadoop.uri?
|
|
||
| private Long getTimeAsMs(String time) { | ||
| if (time == null) { | ||
| LOG.error("Cannot convert to time value.", time); |
There was a problem hiding this comment.
does this print out null properly?
| return refreshInterval; | ||
| } | ||
|
|
||
| private Long getTimeAsMs(String time) { |
There was a problem hiding this comment.
would be good to add some tests for this function
| .schedule(this, getKerberosRefreshInterval(), TimeUnit.MILLISECONDS); | ||
| } else { | ||
| kinitFailCount++; | ||
| LOG.info("runKerberosLogin failed for " + kinitFailCount + " time(s)."); |
| if (path == null || path.trim().length() == 0) { | ||
| absolutePath = fileSystem.getHomeDirectory().toString(); | ||
| } else if (path.startsWith("/")) { | ||
| absolutePath = fileSystem.getHomeDirectory().toString() + path; |
There was a problem hiding this comment.
if a HDFS path starts with /, eg. /root - shouldn't the absolutePath be /root?
| String fileName = f.getPath().getName(); | ||
| if (fileName.startsWith(".") | ||
| || fileName.startsWith("#") | ||
| || fileName.startsWith("~")) { |
There was a problem hiding this comment.
|
How about the compatibility of Hadoop filesystem API against different hadoop versions? |
|
@prabhjyotsingh What's the difference of this PR compared with #2455 ? |
|
@zjffdu yes you are right, I originally thought I have a different idea, but now I see, this will eventually be that. Will close this then. |
4 participants
What is this PR for?
With this PR, I attempt to use Hadoop-filesystem API to implement
NotebookRepo, which in my opinion is very generic in nature, as with this we will be able to support multiple scheme of the URI like local file systemfile:///, Hadoop file systemhdfs://localhost:8020, S3 etc. So far I have tested this only with local and HDFS.What type of PR is it?
[Improvement]
Todos
What is the Jira issue?
How should this be tested?
In zeppelin-site.xml enable these two configs; to write into HDFS.
Screenshots (if appropriate)
Questions: