Hive to Spark Problem

[unknown_user]

I installed spark-job-server and configured the enviroment.config to use yarn-client.

When I run hive-to-spark in knime occurs an error described below:

Execute failed: An error occured: java.lang.IllegalArgumentException: Timeout length must not be negative, question not sent to [Actor[akka://JobServer/user/context-supervisor#1663440268]]

I forgot to set up some argument in enviroment.config or cloudera manager?

[tobias.koetter]

Hi,

please check that the enabled timeouts e.g. yarn-context-creation-timeout in the environment.conf file have valid positive numbers. You can also have a look at the spark job server logs to see if there are more information about the error. The log location is defined in the settings.sh file and are by default LOG_DIR=/var/log/spark-job-server

Bye

Tobias

[unknown_user]

I created a "spark context" before "hive to spark" to test the context. When i tested the enviroment.conf with "local[4]" the context works and the "hive to spark" too.

When i change to yarn-client and uncomment the line yarn-context-creation-timeout, i run knime and the context and "hive to spark" didnt works.

i saw the /var/log/spark-job-server on cluster with tail -f but when i setted yarn-client and run knime, the log didn't have updates.

And knime.log show:

ERROR Create Spark Context 0:24 HTTP Status code: 500 | Response Body: The server was not able to produce a timely response to your request.
ERROR Create Spark Context 0:24 Execute failed: Spark Jobserver gave unexpected response (for details see View > Open KNIME log). Possible reason: Incompatible jobserver version, malconfigured Spark jobserver

[tobias.koetter]

Hi,

if the .log file does not show any messages the job server does not seem to be able to start at all. Can you please have a look into the .out file within the /var/log/spark-job-server directory this might contain some information why the job server can not start. Can you also check if the Job Server is running by opening the Job Server GUI in a browser e.g. :8090.

Bye

Tobias

[unknown_user]

I have the same problem.

The Job server is running, and gui :8090 works fine.

But it does not work with environment.conf set to " yarn-client " + knime environment.

Is there any solutions/suggestions?

if the .log file does not show any messages the job server does not seem to be able to start at all. Can you please have a look into the .out file within the /var/log/spark-job-server directory this might contain some information why the job server can not start. Can you also check if the Job Server is running by opening the Job Server GUI in a browser e.g. :8090.

Bye

Tobias