[SPARK-2460] Optimize SparkContext.hadoopFile api

[scwf]

1 use SparkContext.hadoopRDD() instead of instantiate HadoopRDD directly in SparkContext.hadoopFile.
SparkContext.hadoopRDD() Add necessary security credentials to the JobConf before broadcasting it.

2 broadcast jobConf in HadoopRDD, not Configuration. this will resolve the dead lock issue----
now HadoopRDD broadcast Configuration and in each task (compute method) to get jobConf
then the lock of
conf.synchronized {
val newJobConf = new JobConf(conf)
initLocalJobConfFuncOpt.map(f => f(newJobConf))
HadoopRDD.putCachedMetadata(jobConfCacheKey, newJobConf)
newJobConf
}
will conflict with hadoop version fix Hadoop-10456:

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
synchronized(Configuration.class) {
REGISTRY.put(this, null);
}

we can reproduce the bug like this:
hadoop version 2.4.1
spark master branch

hql("SELECT t1.a, t1.b, t1.c FROM table_A t1 JOIN table_A t2 ON (t1.a = t2.a)")

[AmplabJenkins]

Can one of the admins verify this patch?

[rxin]

Thanks for submitting the pull request.

Did I read this correctly? The master branch deadlocks? If yes, we should file a JIRA for that also and make that more clear. If it is simply about optimizing an API to reduce code, it is a much lower priority issue. If it deadlocks, this needs to be a BLOCKER.

[aarondav]

Is this related to the other conf-related concurrency issue that was fixed recently? #1273

[scwf]

@rxin and @aarondav, yeah ，the master branch deadlocks, it seems locks of #1273 and Hadoop-10456 lead to the problem. when run hivesql self join sql--- hql("SELECT t1.a, t1.b, t1.c FROM table_A t1 JOIN table_A t2 ON (t1.a = t2.a)"), the program stucks.

i think clean SparkContext.hadoopFile api is a better way for fix it. in this way, we do not need the lock in
#1273

[aarondav]

Do we still need this check? I think this may have been the only place we put the JobConf inside the cache.

[scwf]

yeah, there is no need to cache the jobconf if it is in broadcast

[lianhuiwang]

yes,i agree with you. broadcastedConf is cached by blockManager in Broadcast