KNIME 4.1: Keras error ( Selected Keras backend 'Keras(Tensorflow)' is not available anymore

[Jyotendra]

Hi,

I am trying to configure the Keras Network Learner and I am getting an error. Screenshot below.

I have set up the Python and Deep Learning libraries as instructed. When I use the Python Script, I can run the model using Keras. However, it is failing when using Keras network learner. I have also tried to launch Knime using “-clean” mode but without success.

Any suggestion?

Thanks
Jyotendra

[Jyotendra]

One minor edit: title says Python 4.1 - It is actually KNIME 4.1

any help, friends

[HansS]

@Jyotendra changed it to KNIME 4.1

[Jyotendra]

Thanks @HansS

Any pointers on the challenge that I am facing …

[Jyotendra]

I was able to partially solve the issue when I installed Keras on my “Base” environment - Somehow, it was not able to pick up packages from the “Py36_Knime” environment …

[ScottF]

Hi @Jyotendra -

Have you perhaps tried using the New Environment button to create a completely separate environment?

(I can see a warning message that the onnx package is not included in py36_knime, so I’m guessing that’s an environment you’ve been using previously.)

[Jyotendra]

Hi ScottF,

I tried using new environment also - It is not working

One of the reason that I figured out is that in the setting of Python Deep learning, it shows “No environment” when I open it…after few minutes only my conda environment gets loaded. is it possible that Keras doesn’t find any enviorment to work upon?

My error screenshot is

Though my all other python script runs fine.

any idea, why? or what troubleshoot I can run?

[ScottF]

Hi @Jyotendra -

Just to make sure, when you click on the New environment… button on the Python Deep Learning configuration page, you are asked to provide a name for your new environment (by default, I think it’s “py3_knime_dl”) and choose whether you want a CPU or GPU basis.

Did you do this? It takes a few minutes to build the environment, but afterward you should be able to select it from the dropdown. In the screenshot above, you still have no environment selected, so the Keras Network Learner node won’t be able to find it.

Or perhaps you did build the new environment, but for some reason you can’t select anything from the dropdown at all? That would be quite strange.

If you have a little more detail, that would help. Thanks!

[Jyotendra]

Hi @ScottF,

I have also built the environment using New environment. However, after I select the enviornment from the dropdown, press “Apply and close” (or apply), the environment resets to “No Environment” and hence Keras couldn’t find it. It happens every time.

I have created a small screen recorder to show the problem. It is a three minute video but you can skip the part where Keras node is trying to copy the input at port 1.[Two instances - first from 0:05 to 0:46 and another from 1:50 to 3:14)

Let me know if it helps.
Thanks
Jyotendra

[mlauber71]

Setting up deep learning environment with KNIME can sometimes be tricky. You could check these steps and make sure you have the environment running from within python (and also KNIME). Also at the end of the thread there are some hints by @aworker that might or might not be relevant for your case.

Problem with setting a Python Deep Learning Environment

A few more general remarks: with Python and KNIME it is all about compatibility and consistency of the packages. That is where Anaconda tries to help you should not update individual packages but let Anaconda manage that concerning Tensorflow and Keras note which versions are supported by the various KNIME nodes (typically that is not the most recent version) these versions must be compatible with your Python version (it might be that it is to large) I am working on a text about KNIME and Pyt…

And yes I am aware that repeating a task time and time again hoping for a different outcome is sometimes regarded as one sign of madness …

[ScottF]

Hi @Jyotendra -

I tried your link, but it’s just taking me to a general link shortener landing page - no video.

[Jyotendra]

Hi @ScottF,

That was strange - It was working for me 5 hours back and now it not working for me either.

Anyways, here is the link.

[ScottF]

Got it, I see the video now. It clearly demonstrates the issue - thanks for recording it. Let me run this by some other KNIMErs internally and see what I can find out.

[MarcelW]

Hi Jyotendra,

Let me give you some background first:

The Keras Network Learner node performs an integrity check on the Python environment it is supposed to use, before executing the actual model training. This is done to make sure that the required libraries (Keras, TensorFlow, etc.) are properly installed and their versions match the ones that KNIME supports.
This check times out after a certain duration to prevent the node from freezing in case of unexpected external circumstances (e.g., the Python process crashes).

The part “Installation test for Python back end […] timed out” in the error message you experience suggests that such timeout is exactly what happens in your case.
Given that, on the Python Deep Learning preference page in your recording, the steps “Testing Conda installation…”, “Collecting available environments…”, and “Testing Python environment…” take a very long time to complete, I would suspect that the reason for the timeout is that executing conda commands is slow on your machine. This is a known problem on some Windows machines (and presumably for specific Conda versions only) that we have also experienced on some of our internally used machines.

Here is what you can do:

First, try to check whether the problem really is on the Conda side.

1. Open an Anaconda Prompt window. (You should be able to find Anaconda Prompt via the Windows Search/Start Menu if you are unsure where it is located.)
2. Type in and run: conda activate py3_knime_dl
3. Wait for the above command to terminate. If this took a significant amount of time (multiple seconds or more) or the command behaved otherwise unexpectedly, it could be the reason why you experience these timeouts. If not, please let us know, then we will have to look for other possible error sources.

If it is indeed Conda being slow:

1. Add the option -Dknime.dl.installationtesttimeout=3600000 to your knime.ini file (on a new line at the very bottom of the file). You can find the knime.ini in your KNIME installation directory. The option increases the timeout of the integrity check to one hour (3,600,000 milliseconds).
2. Restart KNIME
3. Try to rerun your workflow. The Keras learner node will now be stuck at “Copying input object at port 1” even longer (up to an hour, see above) since that is when the check is carried out.
4. If, after that, the node continues to execute, we have successfully worked around the problem. If not, again, please let us know.

The above is of course just a workaround and waiting for the node to execute for minutes is not ideal. So furthermore you could try to:

• Temporarily deactivate all antivirus software on your computer (including Windows Defender) and see if this improves things. You do this at your own risk, however. This is not a permanent solution but should only be done once to gain more insight into the problem. If it helped, you may want to consider adding exceptions for Conda to the antivirus programs’ filters. Again, at your own risk.
• Downgrade Conda to version 4.7.12. In our experience, this version turned out to work well on some Windows machines on which we previously experienced the problem at hand.

Marcel

[Jyotendra]

Hi Marcel,

Thanks for looking into the problem.
I don’t think Conda is slow because py3_knime_dl gets activated very fast on
I can’t tamper with the antivirus as it is IT controlled so I will try to downgrade Conda to 4.7.12 and will see if it improves.

Jyotendra

[MarcelW]

If downgrading did not help, could you additionally also run the command:

C:\Users\A0262\AppData\Local\Continuum\anaconda3\condabin\conda.bat --version

via the normal Windows Command Line, not the Anaconda Prompt? This is what KNIME actually calls under the hood, maybe its behavior differs from the usual conda command. Does it also run normally/fast?

Marcel

[Jyotendra]

Hi @MarcelW,

By adding the option to Knime.ini file, my problem was solved ( I didn’t do it for 1 hours but for only 5 mins) . I am providing a video for your reference to look at the speed of the execution for other suggestions.
Please let me know if there is anything else that I need to try.

Thanks
Jyotendra

[MarcelW]

Hi Jyotendra,

Happy to hear that at least you’re now able to execute the workflow .

The speed of the command executions in your video indeed seems a bit slow (on my Linux machine, they really complete instantaneously), but not too bad. I’ll double check at our end to see if there’s something else that could further prolongate the delay.

I believe there’s nothing more you could do right now, unfortunately. I hope the current delay when executing a deep learning node is tolerable.

Marcel