Spark Models

Hi, Is it possible to create own Spark Models learner, with customized script code? 


Hi Veronica,

is there a particular Spark model that you need? Feedback on what's missing is always useful for our development priorities.

As for your own Spark model learner nodes you have two options:

The first option is: Use the existing Spark Java Snippet node to learn a model on some data, and then save this model to HDFS. In a second Spark Java Snippet node you can then load the model from HDFS and use it for prediction. It is currently not possible to pass the model itself from one Spark Java Snippet node to the next, because these nodes only have data ports. You could however pass a one-row DataFrame that contains the HDFS path of the saved model. This is not pretty, but it will work.

The second option is: If you want this to be done nicely, it is now possible for you to create your own KNIME nodes! The big data extensions are open source since December. If you are interested in this option briefly drop us an email to to get access to the source code (we are currently still working on publishing the source code).