For some time now we have been trying to connect to the HDFS and create spark context on our Spark cluster with the following workflow:
Connecting to HDFS with kerberos seems to work fine, using HDFS connector (webHDFS with SSL/HttpFS with SSL) or KNOX, yet the Create Spark Context node gives the following error:
ERROR Create Spark Context (Livy) 3:8 Execute failed: Failed to upload Kryo version detector job: Cannot retry request with a non-repeatable request entity. (NonRepeatableRequestException)
We found this error message in the KNIME update-site.zip > org.knime.bigdata.spark.core.livy.source_4.4.0.v202106241517.jar > LivySparkContext.java > 449. line
but couldn’t find what is causing it.
Running the node creates the livy session ID and the workfolder on HDFS for it, but crashes right after.
We overcame a lot of different errors, but got stuck here.
Is this UseCase supported on the Analytics Platform?
Are we missing a configuration?
Hi @botichello and welcome to the KNIME community!
Running Spark via Livy on a secured cluster should work fine. Can you provide some details about your cluster (Type, Cluster Version and Spark Version)? E.g. Cloudera 7.1.7 and Spark 3.0.
The debug logs in KNIME might contain more information. You can enable them in Preferences → KNIME and then select DEBUG as Log Level. Can you post the logs here?
I’m sorry, but KNIME currently only supports Spark up to version 3.0 and you have to select the matching Spark version in the Livy node. We are already working on this and hopefully support more Spark versions in one of the next KNIME releases.
Running this workflow with same configs and certificates to connect the server with 2.4 gives the exact same error log, so it’s surely not about the spark version.
Any other lead for this issue?
and…
May I ask about your roadmap on the implementation of new spark versions?
was this the full error log you posted above? The interesting part in the Stacktrace would be at the end and unfortunately seem to be missing. Maybe the Livy log contains some more information?
Right now we do not support Microsoft SQL Server Big Data Clusters as we never tested it. It seems to use the usual tools like Hadoop/Livy and you might be able to get it running with KNIME. I can’t reproduce this as I don’t have a test SQL Server Big Data Cluster at the moment.
Spark 3.2 support is on the next KNIME summer release roadmap, but it’s not yet finished and plans might change.
the logs above are from the Livy Client / Apache HttpClient that should use the KNIME logger, there is no Spark or Log4j involved at all. If there is nothing more, than the NonRepeatableRequestException seems to be the last mentioned problem. I have never seen such error before and can’t reproduce this right now. You might try to look into the Cluster logs and check KNOX, Livy and YARN container logs. Maybe its an permission problem and KNOX or Livy fails somewhere.