DB Loader not working with Snowflake driver from 3.17

KNIME Analytics Platform
,
1

Hi,

I was trying to update Snowflake driver, so it could read vector tables.

According to
JDBC Driver release notes for 2024 | Snowflake Documentation
, vector support is supported from version 3.16.0

I upgraded to version 3.20, by downloadning driver via Downloading / integrating the JDBC Driver | Snowflake Documentation.

With version 3.20, I could read Vector table.
But unfortunately, the node “DB Loader” stopped working.

I would fail with message:

Remote file ‘users/3715437609/knime2dbf2211f05e9214d8e.csv.gz’ was not found. If you are running a copy command, please make sure files are not deleted when they are being loaded or files are not being loaded into two different tables concurrently with auto purge option.

Then I tried a range of drivers:

Modifying C:\Users\tesc\knime-workspace\.metadata\.plugins\org.eclipse.core.runtime\.settings\org.knime.database.prefs

With content

drivers/snowflake_test_3.14.1/name=Snowflake Test 3.14.1
drivers/snowflake_test_3.14.1/database_type=snowflake
drivers/snowflake_test_3.14.1/description=Snowflake JDBC Driver Test
drivers/snowflake_test_3.14.1/driver_class=net.snowflake.client.jdbc.SnowflakeDriver
drivers/snowflake_test_3.14.1/paths/0=C:\\KNIME\\snowflake_test\\snowflake-jdbc-3.14.1.jar
drivers/snowflake_test_3.14.1/url_template=jdbc\:snowflake\://novozymes-prod.snowflakecomputing.com/?warehouse\=<warehouse>&role\=[role]&db\=[database]&schema\=[schema]
drivers/snowflake_test_3.14.1/version=3.14.1
drivers/snowflake_test_3.16.1/name=Snowflake Test 3.16.1
drivers/snowflake_test_3.16.1/database_type=snowflake
drivers/snowflake_test_3.16.1/description=Snowflake JDBC Driver Test
drivers/snowflake_test_3.16.1/driver_class=net.snowflake.client.jdbc.SnowflakeDriver
drivers/snowflake_test_3.16.1/paths/0=C:\\KNIME\\snowflake_test\\snowflake-jdbc-3.16.1.jar
drivers/snowflake_test_3.16.1/url_template=jdbc\:snowflake\://novozymes-prod.snowflakecomputing.com/?warehouse\=<warehouse>&role\=[role]&db\=[database]&schema\=[schema]
drivers/snowflake_test_3.16.1/version=3.16.1
drivers/snowflake_test_3.17.0/name=Snowflake Test 3.17.0
drivers/snowflake_test_3.17.0/database_type=snowflake
drivers/snowflake_test_3.17.0/description=Snowflake JDBC Driver Test
drivers/snowflake_test_3.17.0/driver_class=net.snowflake.client.jdbc.SnowflakeDriver
drivers/snowflake_test_3.17.0/paths/0=C:\\KNIME\\snowflake_test\\snowflake-jdbc-3.17.0.jar
drivers/snowflake_test_3.17.0/url_template=jdbc\:snowflake\://novozymes-prod.snowflakecomputing.com/?warehouse\=<warehouse>&role\=[role]&db\=[database]&schema\=[schema]
drivers/snowflake_test_3.17.0/version=3.17.0
drivers/snowflake_test_3.20.0/name=Snowflake Test 3.20.0
drivers/snowflake_test_3.20.0/database_type=snowflake
drivers/snowflake_test_3.20.0/description=Snowflake JDBC Driver Test
drivers/snowflake_test_3.20.0/driver_class=net.snowflake.client.jdbc.SnowflakeDriver
drivers/snowflake_test_3.20.0/paths/0=C:\\KNIME\\snowflake_test\\snowflake-jdbc-3.20.0.jar
drivers/snowflake_test_3.20.0/url_template=jdbc\:snowflake\://novozymes-prod.snowflakecomputing.com/?warehouse\=<warehouse>&role\=[role]&db\=[database]&schema\=[schema]
drivers/snowflake_test_3.20.0/version=3.20.0

image
image1502×556 43.8 KB

Version 3.16.1 seems to be the last version that works with “DB Loader” node.
3.17.0 fails with Execute failed: Could not initialize class net.snowflake.client.jdbc.SnowflakeFileTransferAgent

2

I found this on snowflake community page

Remote file was not found when loading data into Snowflake Table from an Internal Stage

3

I tried looking in snow flake log

select q.START_TIME, q.EXECUTION_STATUS, q.ERROR_MESSAGE, q.QUERY_TEXT--, q.*
from table(information_schema.query_history()) AS q
WHERE QUERY_TYPE = 'PUT_FILES'
AND USER_NAME = CURRENT_USER
order by q.START_TIME desc;

But that looks fine

image
image1542×218 16.2 KB

4

It’s the PUT commands, that no longer works

OK with 3.16.1

image
image1041×579 36.9 KB

Not OK with 3.20.1
No file transferred with PUT

image
image1079×566 36.7 KB

5

Reported to snowflake github issues tracker
PUT command does not transfer file for JDBC driver from 3.17.0 · Issue #1984 · snowflakedb/snowflake-jdbc

1 Like
6

There is not a bug in the Snowflake driver.

There seems to be a bug in KNIME “DB Loader” node, that prepends an “/” too much, in file:///C

image
image1766×241 21.2 KB

PUT 'file:///C:/Users/tesc/AppData/Local/Temp/knime_Snowflake Drive_17935/dbLoader-2514/knime2db605fcdead0574dd0.csv.gz' '@~' SOURCE_COMPRESSION=GZIP AUTO_COMPRESS=FALSE

The extra / was apparently accepted as input until 3.16.1

For reference, snowflake documentation on PUT
PUT | Snowflake Documentation

PUT ‘file://C:/temp/load data/mydata.csv’ @~ AUTO_COMPRESS=TRUE;

7

Hello @tescnovonesis ,
thanks a lot for reproducing the error in such detail and filing the bug in Github. Using the three / in the beginning of the URI is the proper URI format for local Windows paths as described here. I have added a comment about this to your bug in Github and hope that this gets rather fix in the driver. If they will not change this odd behavior in the driver we plan to release a fix with the 5.4.1 and 5.3.4 that will implement a special path conversion for the Snowflake driver.
Bye
Tobias

2 Likes