Setup spark-job-server

Hi all,

I'm totally newbie trying to setup spark-job-server-0.6.2.2-KNIME_cdh-5.4.tar.gz on a namenode (also spark master) from an openstack cluster with plugin Spark 1.6 ( standalone with 1 master and 3 workers). To check for cloudera hadoop version:

user@ubuntu:~$ hadoop version

Hadoop 2.6.0-cdh5.4.0 Subversion http://github.com/cloudera/hadoop -r c788a14a5de9ecd968d1e2666e8765c5f018c271 Compiled by jenkins on 2015-04-21T19:17Z Compiled with protoc 2.5.0 From source with checksum cd78f139c66c13ab5cee96e15a629025 This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.4.0.jar

Hive is not installed by default.

Following instruction to install KNIME Spark Executor 1.7.2, it appeared the spark-job-server's not successfully started. Below is the log content:

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 Uncaught error from thread [Uncaught error from thread [JobServer-akka.actor.default-dispatcher-6] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer-akka.actor.default-dispatcher-4] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer] java.lang.AbstractMethodError at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) at akka.actor.ActorCell.invoke(ActorCell.scala:487) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) at akka.dispatch.Mailbox.run(Mailbox.scala:220) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) JobServer] java.lang.AbstractMethodError at akka.actor.ActorCell.create(ActorCell.scala:580) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I did google the error and found somethings related to the version conflict but not sure if it's the real cause or not and how to fix it.

Any help will be appreciated. Thanks.

Hi,
this error indicates an incompatibility problem with the Spark version and your Cloudera cluster. Cloudera is using an older version of Akka then the standard Spark installation. That is why you need to make sure to use the Spark installation files from Cloudera. Are you able to start Spark jobs via the Spark shell?
Bye
Tobias