Skip to content

Magister89/spark-yarn-client

Repository files navigation

spark-shell --jars com.brain.yarn-spark-client-assembly-0.1.jar \
--files \
core-site.xml,\
hadoop-env.sh,\
hdfs-site.xml,\
hive-env.sh,\
hive-site.xml,\
mapred-site.xml,\
ssl-client.xml,\
yarn-site.xml,\
user.keytab

import com.brain.yarnsparkclient._
val conf = new YarnConfiguration
conf.init()
val ugi = new KerberosAuthentication("[email protected]", "datalabs.keytab", conf).getUgi()
val yarnClient = new YarnKerberosClient(conf, ugi, spark)
yarnClient.init()
yarnClient.start()
val ds = yarnClient.getApplications
val types = Set("SPARK")
val dsFiltered = yarnClient.getApplications(types)
ds.show()
dsFiltered.show()

About

Yarn Client implemented in Scala/Spark

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages