Spark Number To Category (Apply) : 3:208 : Execute failed: An error occured: akka.pattern.AskTimeoutException: Timed out

KNIME Analytics Platform
1

Getting following time out error. I have increased all the timeout parameter in spark job server.

Not clear where the time out comming from. This error occurs in 5 seconds. Please let me know what time out parameter to change.

Thanks,

Prakash

 

 

2017-07-14 11:49:44,161 : DEBUG : KNIME-Worker-16 : Spark Number To Category (Apply) : Spark Number To Category (Apply) : 3:208 : reset

2017-07-14 11:49:44,161 : DEBUG : KNIME-Worker-16 : SparkNodeModel : Spark Number To Category (Apply) : 3:208 : In reset() of SparkNodeModel. Calling deleteRDDs.

2017-07-14 11:49:44,161 : ERROR : KNIME-Worker-16 : Spark Number To Category (Apply) : Spark Number To Category (Apply) : 3:208 : Execute failed: An error occured: akka.pattern.AskTimeoutException: Timed out (for details see View > Open KNIME log)

2017-07-14 11:49:44,162 : DEBUG : KNIME-Worker-16 : Spark Number To Category (Apply) : Spark Number To Category (Apply) : 3:208 : Execute failed: An error occured: akka.pattern.AskTimeoutException: Timed out (for details see View > Open KNIME log)

com.knime.bigdata.spark.core.context.jobserver.request.RestoredThrowable: akka.pattern.AskTimeoutException: Timed out

at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)

at akka.actor.Scheduler$$anon$11.run(Scheduler.scala:118)

at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)

at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)

at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:455)

at akka.actor.LightArrayRevolverScheduler$$anon$12.executeBucket$1(Scheduler.scala:407)

at akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:411)

at akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)

at java.lang.Thread.run(Thread.java:745)

2

Hi,

this may not actually be solvable by increasing timeouts. AskTimeoutExceptions in jobserver are usually indicative of (a) Jobserver misconfiguration, or (b) unusally high load on the machine where jobserver runs or (c) problems in the Hadoop cluster.

You may find it useful to look at the Spark jobserver logfiles.

If you have set context-per-jvm = true in jobserver's environment.conf there are two logfiles that are of interest:

  • /var/log/spark-job-server/spark-job-server.log
  • /var/log/spark-job-server/<some-context-identifier>/spark-job-server.log

If you have set context-per-jvm = false in jobserver's environment.conf everything will be just in /var/log/spark-job-server/spark-job-server.log.

- Björn