I am using Create DataBricks env node to connect KNIME and databricks. It was working fine, I was able to connect to databricks and do some execution. But now I am not able to connect to databricks and getting below error
KNIME is using that REST API to connect to the Databricks platform. There is no workaround, but we will squeeze in a fix for KNIME Analytics Platform 4.3.2 (scheduled for next week).
Having said that, there is a discussion for us to be had with the Databricks people here. Changing a REST API without prior notice and without bumping the REST API version is questionable practice for a cloud service. It has high risk of breaking consumers of the API, which may rely on it for their critical workloads.
It’s really annoying Databrick’s did this without warning, and can imagine that it’s caused all sorts of problems! @bjoern.lohrmann do you know which day 4.3.2 is likely to be released? I’m desperate for this fix!!
But be aware that the nightly build may or may not have other issues. It reflects the current development status for the next KNIME version and has not been fully tested.
Excellent, thanks for the info @bjoern.lohrmann. I’m glad that Knime are reacting to the issue quickly
May try out the nighly-build to get me going in the meantime…
Thanks
Iain
@bjoern.lohrmann I am getting an error as well with knime4.3.1 – Do you think this is the same issue? This is my FIRST time trying to connect –
ERROR Create Databricks Environment 0:1 Execute failed: Problem with reading the data, class org.knime.bigdata.databricks.rest.dbfs.FileInfo, ContentType: application/json.
We released KNIME 4.2.5 that contains a fix to handle the changes in the Databricks REST API that should fix the problem. We are working on a 4.3 bug fix release and hopefully releasing it soon.
@webstar@ijsmiller Due to some unforeseen issues (not Databricks-related) we had to postpone the 4.3.2 bugfix release. Currently, we plan to release it on Monday.
Until then, there are 4.2.5 (as Sascha mentioned) and the night build which contain the Databricks bugfix.
Hi @sascha.wolke - I tested this in the Nightly build, and it was fine. However, I’ve finally updated to 4.3.2, and I get the same error as before when using the “Create Databricks Environment (deprecated)” tool. When using the updated tool “Create Databricks Environment”, I am now getting the exact same error that @webstar reported…
Version of “KNIME Databricks Integration” on my main installation is 4.3.2.v202103021016
Version on the Nightly build I’ve been using is 4.4.0.v202102261546
not sure what goes wrong updating some installations. You can download a fresh 4.3.2 version instead of updating or do the following:
Open the knime.ini in a simple text editor, add -clean in the first line and restart KNIME. (Be sure to remove it afterward as it makes the startup time longer.)
Update: Helps and the -clean option must be the first option in the knime.ini
we try to fix the update problem, but can’t reproduce it. Do you still have the updated KNIME installation with the broken Create Databricks Environment node?
If yes, can you run KNIME from the commandline with the parameter -console. This should start KNIME as usual and you can select your workspace. Run the node and make sure it is still broken with the error above. Than go back to the commandline promt and press enter. Now you should see the osig> promt. After entering b org.knime.bigdata.databricks you should see a lot of data about the databricks plugin. It would be helpful if you can add this here. Just mark everything and than copy+past it in some text editor.
Hi @sascha.wolke
I did a clean restart of Knime, which resolved the issue.
NB I needed to put -clean at the start of knime.ini, above -startup option, and not below…
Thanks very much for your help
Iain
Thank you @ijsmiller , I am downloading 4.3.2 now and hopefully will have similar success; Just curious if you are access databricks [outside of knime] with active directory login? Did you make any special changes ?
@webstar Glad your connection to Databricks is now working. In answer to your question, I can’t use AAD login unfortunately, as “Knime” has not been permitted to authenticate on our Azure environment. Therefore I use a token…
