KNIME capability to access spark using Livy

[ashu9719]

Hi

I am trying to connect spark running on AWS EMR 5.11 using KNIME.

EMR does not have JobServer configured hence trying to connect using Livy url (available in EMR 5.11) but it is not working.

Is there any way to use Livy URL to connect with Spark on EMR. OR some other way to connect spark running on EMR.

Error :- 

ERROR Database Connector   2:1        Execute failed: Could not create connection to database: connect timed out
ERROR Create Spark Context 0:14       HTTP Status code: 404 | Response Body: 
ERROR Create Spark Context 0:14       Execute failed: Spark Jobserver gave unexpected response (for details see View > Open KNIME log). Possible reason: Incompatible Jobserver version, malconfigured Spark Jobserver

Thanks in Advance

[bjoern.lohrmann]

Hi,

unfortunately we currently do not support the Livy jobserver. Support for Livy is in the works and we will likely have a first version available for the summer release.

Currently, we only support "Spark Jobserver", which needs to be installed on a cluster edge node. On our website we provide an Spark Jobserver binaries that have been tested and verified to work with KNIME, as well as an installation guide (PDF):

https://www.knime.com/knime-extension-for-apache-spark#install

 

Best,

Björn

[ashu9719]

Thanks a lot for the reply and support.

[ashu9719]

It works :)