KNIME integration with HPC resource managers such as LSF, SLURM and other

KNIME Analytics Platform
1

Hi,

I'm new to KNIME, and I have a faculty member who would like to use KNIME and Hadoop.  Currently, we have a high performance computing cluster running Spark and eventually Hadoop.  Our cluster is using IBM's LSF, that facilitates Spark and Hadoop job submission as parrallel jobs.  I would like to know how I can use KNIME server to with the cluster, as in my current setup KNIME would not be connecting to Hadoop for querry submission, these querries would need to go through LSF.  Basically, are there any integration plugins that KNIME has for integrating with Cluster resource managers?

Thank you,

Semir​

2

Hi Semir,

the only cluster integration that is currently available is for the sun grid engine (and this one is commercial). Starting with the next release this summer the cluster integration framework will be released as open source. This enables the community to develop custom integrations. Together with the summer release their will also be a Unicore integration which itself can be installed in the cluster to communicate to LSF. We plan to add more integrations but have no specific plans yet.

Cheers, Patrick