I am using Create DataBricks env node to connect KNIME and databricks. It was working fine, I was able to connect to databricks and do some execution. But now I am not able to connect to databricks and getting below error
ERROR Create Databricks Environment 7:1 Execute failed: Unrecognized field “modification_time” (class org.knime.bigdata.databricks.rest.dbfs.FileInfo), not marked as ignorable (3 known properties: “is_dir”, “file_size”, “path”])
at [Source: (SequenceInputStream); line: 1, column: 119] (through reference chain: org.knime.bigdata.databricks.rest.dbfs.FileInfo[“modification_time”]) (UnrecognizedPropertyException)
Can someone let me know how to resolve this issue.
it seems that Databricks has changed its REST API just a few days ago:
KNIME is using that REST API to connect to the Databricks platform. There is no workaround, but we will squeeze in a fix for KNIME Analytics Platform 4.3.2 (scheduled for next week).
Having said that, there is a discussion for us to be had with the Databricks people here. Changing a REST API without prior notice and without bumping the REST API version is questionable practice for a cloud service. It has high risk of breaking consumers of the API, which may rely on it for their critical workloads.
It’s really annoying Databrick’s did this without warning, and can imagine that it’s caused all sorts of problems!
@bjoern.lohrmann do you know which day 4.3.2 is likely to be released? I’m desperate for this fix!!
@ijsmiller Hard to say which day precisely. Probably until Thursday, depending on how the release testing is going.
Be aware that if it’s very urgent you can also use our nightly build:
But be aware that the nightly build may or may not have other issues. It reflects the current development status for the next KNIME version and has not been fully tested.
Excellent, thanks for the info @bjoern.lohrmann. I’m glad that Knime are reacting to the issue quickly
May try out the nighly-build to get me going in the meantime…
@bjoern.lohrmann I am getting an error as well with knime4.3.1 – Do you think this is the same issue? This is my FIRST time trying to connect –
ERROR Create Databricks Environment 0:1 Execute failed: Problem with reading the data, class org.knime.bigdata.databricks.rest.dbfs.FileInfo, ContentType: application/json.
We released KNIME 4.2.5 that contains a fix to handle the changes in the Databricks REST API that should fix the problem. We are working on a 4.3 bug fix release and hopefully releasing it soon.
@webstar Yes, this is the same error.
@webstar @ijsmiller Due to some unforeseen issues (not Databricks-related) we had to postpone the 4.3.2 bugfix release. Currently, we plan to release it on Monday.
Until then, there are 4.2.5 (as Sascha mentioned) and the night build which contain the Databricks bugfix.
@ijsmiller @webstar We released KNIME 4.3.2 yesterday.
Hi @sascha.wolke - I tested this in the Nightly build, and it was fine. However, I’ve finally updated to 4.3.2, and I get the same error as before when using the “Create Databricks Environment (deprecated)” tool. When using the updated tool “Create Databricks Environment”, I am now getting the exact same error that @webstar reported…
Version of “KNIME Databricks Integration” on my main installation is 4.3.2.v202103021016
Version on the Nightly build I’ve been using is 4.4.0.v202102261546
not sure what goes wrong updating some installations. You can download a fresh 4.3.2 version instead of updating or do the following:
Open the knime.ini in a simple text editor, add -clean in the first line and restart KNIME. (Be sure to remove it afterward as it makes the startup time longer.)
Update: Helps and the -clean option must be the first option in the knime.ini
we try to fix the update problem, but can’t reproduce it. Do you still have the updated KNIME installation with the broken Create Databricks Environment node?
If yes, can you run KNIME from the commandline with the parameter
-console. This should start KNIME as usual and you can select your workspace. Run the node and make sure it is still broken with the error above. Than go back to the commandline promt and press enter. Now you should see the
osig> promt. After entering
b org.knime.bigdata.databricks you should see a lot of data about the databricks plugin. It would be helpful if you can add this here. Just mark everything and than copy+past it in some text editor.
Thanks for the advice, I will try the restart with -clean option you suggested.
In the meantime, I have captured the data you requested, using both the “Create Databricks Environment”, and the deprecated version of that.
Please see attached
Iain2021-03-17 Create Databricks Environment.txt (183.6 KB) 2021-03-17 Create Databricks Environment (deprecated).txt (184.0 KB)
I did a clean restart of Knime, which resolved the issue.
NB I needed to put -clean at the start of knime.ini, above -startup option, and not below…
Thanks very much for your help
Thank you @ijsmiller , I am downloading 4.3.2 now and hopefully will have similar success; Just curious if you are access databricks [outside of knime] with active directory login? Did you make any special changes ?
I can now confirm my ability to connect and use my databricks environment
Hi @ijsmiller, thanks for the OSGI output and the hint to put the clean option at the start!
@webstar Glad your connection to Databricks is now working. In answer to your question, I can’t use AAD login unfortunately, as “Knime” has not been permitted to authenticate on our Azure environment. Therefore I use a token…
Yes token works for me as well
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.