Can we run PySpark code through Knime

KNIME Extensions
,
1

Hi Team,

I have a python file in databricks repos and want to execute it on my local machine in knime.

I was able to connect to databrick using databricks file system connector but I am not sure if I can execute a python script located in databricks repo in knime.

Any help would be appreciated!

Thank you
Srinivas

2

This should be possible but you might try a few things with regards to the user rights and repositories.

Here is one example from the hub where a spark connection is being established with Databricks

Here is an example with various operations on a big data cluster also with some Pyspark examples.

Though this is a different system the principles should be the same. Question might be how to tell the spark environment where your code is.

I have an example how to import and integrate Python code from a local machine, not sure if something similar could be done on a Databricks system.

5 Likes
3

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.