Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[KYUUBI-133]fix #133 token expiration in HadoopRDD getPartitions #134

Merged
merged 3 commits into from
Dec 20, 2018
Merged

[KYUUBI-133]fix #133 token expiration in HadoopRDD getPartitions #134

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4b7f2c5
[KYUUBI-133]fix #133 token expiration in HadoopRDD getPartitions
yaooqinn Dec 13, 2018
40ac044
turn off union rdd parallel lising
yaooqinn Dec 20, 2018
203f615
rm unrelated change
yaooqinn Dec 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions kyuubi-server/src/main/scala/org/apache/spark/KyuubiSparkUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ object KyuubiSparkUtil extends Logging {
val HDFS_CLIENT_CACHE_DEFAULT = "true"
val FILE_CLIENT_CACHE: String = SPARK_HADOOP_PREFIX + "fs.file.impl.disable.cache"
val FILE_CLIENT_CACHE_DEFAULT = "true"
val RDD_PAR_LISTING: String = SPARK_PREFIX + "rdd.parallelListingThreshold"

// Runtime Spark Version
val SPARK_VERSION: String = org.apache.spark.SPARK_VERSION
Expand Down Expand Up @@ -282,6 +283,14 @@ object KyuubiSparkUtil extends Logging {
if (UserGroupInformation.isSecurityEnabled) {
conf.setIfMissing(HDFS_CLIENT_CACHE, HDFS_CLIENT_CACHE_DEFAULT)
conf.setIfMissing(FILE_CLIENT_CACHE, FILE_CLIENT_CACHE_DEFAULT)
// If you are using Kyuubi against kerberized HDFS, you will run into HDFS_DELEGATION_TOKEN
// expiration in some particular sql queries. This exception is usually caught in
// HadoopRDD.getPartitions, where the JobConf has no Credentials because it is generated by
// Configuration, and the UGI.getCurrentUser contains only the oldest tokens which are
// destined to expire. The reason seems to be parallel listing UnionRDD's sub RDDs using a
// ForkJoinPool which points to another calling context. Turn off parallel listing seems
// to be a solution to this issue.
conf.setIfMissing(RDD_PAR_LISTING, Int.MaxValue.toString)
}
}

Expand Down