Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

can HadoopSparkJob support spark job running on mesos? #257

Open
wangqiaoshi opened this issue Feb 6, 2017 · 1 comment
Open

can HadoopSparkJob support spark job running on mesos? #257

wangqiaoshi opened this issue Feb 6, 2017 · 1 comment

Comments

@wangqiaoshi
Copy link

I don't need spark job running on yarn,but running on mesos.
now can HadoopSparkJob support it?
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/conf/YarnConfiguration at azkaban.jobtype.HadoopSparkJob.sparkJarsHelper(HadoopSparkJob.java:397) at azkaban.jobtype.HadoopSparkJob.testableGetMainArguments(HadoopSparkJob.java:342) at azkaban.jobtype.HadoopSparkJob.getMainArguments(HadoopSparkJob.java:309) at azkaban.jobExecutor.JavaProcessJob.createCommandLine(JavaProcessJob.java:65) at azkaban.jobExecutor.JavaProcessJob.getCommandList(JavaProcessJob.java:54) at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:93) at azkaban.jobtype.HadoopSparkJob.run(HadoopSparkJob.java:226) at azkaban.execapp.JobRunner.runJob(JobRunner.java:693) at azkaban.execapp.JobRunner.run(JobRunner.java:545) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.conf.YarnConfiguration at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 14 more

@wangqiaoshi
Copy link
Author

wangqiaoshi commented Feb 6, 2017

I found import org.apache.hadoop.yarn.conf.YarnConfiguration in HadoopJobUtils class.
i add hadoop-yarn-api-2.7.2.jar in jobtype.global.classpath,i will not throw YarnConfiguration Class not found error Exception

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant