Issue with Retrieving Data from Databricks Delta Lake to KNIME via JDBC

[mertyavuz]

Hi Support Team,

I am trying to establish a connection from Databricks Lakehouse (Delta Lake) using the KNIME tool and am encountering issues in retrieving data. Here are the steps I followed:

1. Installed the Databricks JDBC driver and configured a connection in KNIME using the “DB Connector” node.
   Status: Connection to Delta Lake was successful.
2. Added the “DB Table Selector” node to browse the metadata.
   Status: Metadata (schemes and tables) was visible, and I was able to select tables and write queries. The node executed successfully.
3. Used the “DB Reader” node to retrieve data into KNIME Desktop.
   Issue: Although column names are displayed, the data itself is not retrieved. Instead, I see the following error message:
   this optional feature.

It seems that data retrieval from Delta Lake is either not supported or misconfigured, or there might be compatibility issues between KNIME and Databricks JDBC for Delta tables.

Request:
Could you please advise on the best way to work with Delta tables in KNIME? If there is a recommended configuration or a specific workaround to retrieve data into KNIME Desktop, I would appreciate guidance on that.

Thank you for your assistance.

[tobias.koetter]

Hello @mertyavuz ,

please disable transactions via the Advanced tab of the DB Connector node since they are not supported by Databricks Warehouses

Ideally you install the KNIME Databricks Integration which will come with the dedicated Databricks type and required JDBC driver which do this under the hood. Once installed you DB Connector should look like this

With the next release of the KNIME Analytics Platform (version 5.4) the Databricks extension will include a dedicated Databricks Warehouse Connector node which will make connecting even easier.

Bye
Tobias

[mertyavuz]

Hi @tobias.koetter ,
Thank you for your respond. Issue is solved once I use database driver which I installed from “Knime Databricks Integration” as you addressed.

BTW, In my advanced tab, there is no Transaction section. But I did not need anyway…