[ZEPPELIN-1587] (WIP) Add impersonation routine in SparkInterpreter for current user

[khalidhuseynov]

What is this PR for?

This is to add impersonation routine for SparkInterpreter, meaning any communication with hadoop hdfs should be done with current user credentials

What type of PR is it?

Improvement

Todos

• - add privilege mode
• - add tests

What is the Jira issue?

ZEPPELIN-1587

How should this be tested?

executing hdfs related file write should be done by your logged in username, e.g. in the example i used the following code:

val file = sc.textFile("hdfs:/inputFilePath")
file.cache()
val wordCount = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)
wordCount.saveAsTextFile("hdfs:/outputFolderPath")

Screenshots (if appropriate)

Questions:

• Does the licenses files need update? no
• Is there breaking changes for older versions? no
• Does this needs documentation? possibly

[zjffdu]

I don't know how much this helps in impersonation. At least it doesn't affect executor. Do you see any impact on the driver side ?

[khalidhuseynov]

@zjffdu i updated description and pushed some changes. as you can see from screenshot it makes usage of current username when writing to hdfs. So in the example above, although out_user1 folder is owned by user1, still it contains inside files (e.g. part0000) with different owner khalidhuseynov. So my guess is that it affects only driver and not the executors, and executors are the ones writing part0000. let me know what you think on it.
also feedback from @Leemoonsoo @prabhjyotsingh @felixcheung would be appreciated on this one as well

[Leemoonsoo]

Using --proxy-user instead of doAs would work? https://issues.apache.org/jira/browse/ZEPPELIN-1730

[khalidhuseynov]

@Leemoonsoo yes that's possible; moreover I originally implemented it using proxy-user. the only thing is that your hdfs for example in that case should be configured with

<property> 
<name>hadoop.proxyuser.<username>.hosts</name> 
<value>*</value> 
</property> 
 
<property> 
<name>hadoop.proxyuser.<username>.groups</name> 
<value>*</value> 
</property> 

in hdfs-site.xml so that you can't run without that configuration, which makes sense.

[zjffdu]

@khalidhuseynov I think proxy-user is the correct way to implement impersonation for spark. Your approach in this PR only apply to driver side. If you look at the RM UI, the yarn app owner is the still the origin user. The reason that the owner of file is user1 in your screenshot is that the hdfs commit operation is invoked in driver side. And the change in hdfs-site.xml is necessary for impersonation.

[khalidhuseynov]

@zjffdu makes sense. then i'll change it back to use proxy-user with additional flag, in order not to affect first time users not using impersonation.

[astroshim]

@khalidhuseynov How about using System.setProperty("HADOOP_USER_NAME", getUserName()); instead of proxy-user setting?
Could you review astroshim#15?
It might work as you expected.

[khalidhuseynov]

@astroshim thanks for help, I'll test with proxy-user first since it's most recommended way, and possibly we can use your configuration there.

[khalidhuseynov]

So I've done some research and --proxy-user argument is used with either spark-submit or spark-shell normally, and each of them creates new spark context. The problem is that Zeppelin calls spark-submit only once in shared and scoped modes when initializing interpreter, and afterwards single spark context is used. Thus it makes having impersonation through --proxy-user on spark interpreter in shared and scoped mode complicated. For isolated mode, it's easier since we can pass user with SparkConf for each spark context on initialization. any suggestions /opinions are welcome.

and thanks @astroshim for PR and help, that would work in isolated mode.

[zjffdu]

That's correct, impersonation for spark interpreter can only be applied to isolated mode. It is due to capability of spark.

[astroshim]

I agree with @khalidhuseynov's opinion.
I was keep wondering it's possible impersonation on shared or scoped mode.
My PR is just created for the testing and only tested on 'yarn' mode so @khalidhuseynov please just refer to it.