Unable to set up Python Deep Learning Environemnt

[Mark_Zheng]

Dear All,

I have been trying to replicate the workflow DBPedia texts classification with BERT by Redfield – KNIME Hub with my own data. However, as I was setting up deep learning environment, I found the Keras and Tensorfow requirements appear to be self-contradictory. I tried many other ways (e.g., updating tensorflow, downgrading python, updating Keras) but I just couldn’t find the right combination of versions for Keras, python, and tensorflow. Anyone has an idea?

[mlauber71]

@Mark_Zheng Welcome to the Knime Forum.

You might want to start by reading the Python guide
https://docs.knime.com/latest/python_installation_guide/index.html

And then the.
https://docs.knime.com/latest/deep_learning_installation_guide/index.html

I have once tried to condense that into a short story, you might have to switch the versions to the current ones

Problem with setting a Python Deep Learning Environment

A few more general remarks: with Python and KNIME it is all about compatibility and consistency of the packages. That is where Anaconda tries to help you should not update individual packages but let Anaconda manage that concerning Tensorflow and Keras note which versions are supported by the various KNIME nodes (typically that is not the most recent version) these versions must be compatible with your Python version (it might be that it is to large) I am working on a text about KNIME and Pyt…

It is about following the right combination of versions and sometime even the conda environments used seem to play a role. So you might have to experiment with channel priorities (might depend on your operating system)

[Daniel_Weikert]

Ths is an interesting topic.
Could it be that KNIME is lagging in terms of compatablility with the latest packages (e.g tensorflow)?
So far I always went back and used a python scripting node instead of the Tensorflow or keras nodes provided.
Just want to check, maybe I am wrong.
Thanks

[MarcelW]

Hi @Mark_Zheng,

The sections Keras and TensorFlow 2 in your screenshot refer to different versions of Keras/TensorFlow. The Keras section refers to old versions of the Keras library (versions 2.2.x and below) that can be used with TensorFlow 1 but are not compatible with TensorFlow 2. Since then, the original Keras library has been discontinued and replaced by tf.keras which is a part of TensorFlow 2 itself. (For more details, please refer to Keras’ version history.)
So the relationship between the different versions of Keras and TensorFlow is a bit confusing. And admittedly, the current layout of our Python Deep Learning Preferences page does not really help to clarify things. We plan to address this in the future.

The BERT nodes are based upon TensorFlow 2, so only the bottom section is relevant to you. You can use the New environment… button there to create a new generic TensorFlow 2 environment. The BERT nodes however require some BERT-specific additional packages that are not part of that generic environment (see “Required Python packages” on the KNIME Hub page of the workflow for the list of packages). You will need to install these packages manually in the created environment (e.g. via the Anaconda Navigator or the Anaconda Prompt command line).

Marcel

[MarcelW]

Hi @Daniel_Weikert,

The TensorFlow nodes only support TensorFlow 1 (and should be renamed accordingly). The same is true for the Keras nodes (see my post above).

There are the TensorFlow 2 nodes but their support also lags some versions behind (we currently test and therefore recommend to use TensorFlow v2.2.0 internally, whereas the most recent version of TensorFlow is v2.4.2). Also there are no training or layer nodes available at the moment, so currently you need to resort to scripting when you want to train TensorFlow 2 based models, yes.

Marcel

[Mark_Zheng]

Dear MarceIW,

Thanks for the help! I have tried to focus on fixing the Tensorflow 2 part. I created an environment using the “new environemnt” function. But the notification said I will need a tensorflow version from 2.2.0 to 3.0.0.

When I tried to upgrade my tensorflow to 2.3.0 in my anaconda 3, I got the follow error message as follows:
UnsatisfiableError: The following specifications were found

to be incompatible with the existing python installation in your environment:

Specifications:

- tensorflow==2.3.0 → python[version='3.7.|3.8.’]

Your python: python=3.6

If python is on the left-most side of the chain, that’s the version you’ve asked for.

When python appears to the right, that indicates that the thing on the left is somehow

not available for the python version you are constrained to. Note that conda will not

change your python version to a different minor version unless you explicitly specify

that.

The following specifications were found to be incompatible with each other:

Output in format: Requested package → Available versions

Package tensorflow-base conflicts for:

tensorflow==2.3.0 → tensorflow-base==2.3.0[build=‘mkl_py37h7075554_0|eigen_py38h75a453f_0|mkl_py38ha8c2253_0|eigen_py37h17acbac_0|gpu_py37h18d21e4_0|gpu_py38h7339f5a_0’]

tensorflow==2.3.0 → tensorflow-estimator[version=’>=2.3.0’] → tensorflow-base[version=’>=2.3.0,<2.4.0a0|>=2.4.1,<2.5.0a0’]

Package requests conflicts for:

python=3.6 → pip → requests

tensorflow==2.3.0 → tensorboard[version=’>=2.3.0’] → requests[version=’>=2.21.0|>=2.21.0,<3’]

Alternatively, when I tried to install tensorflow 2.3.0 from my anaconda command, I got this:
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x000002A3055E02E0>, ‘Connection to pypi.org timed out. (connect timeout=15)’)’: /simple/tensorflow/
Attempts to upgrade to python 3.7 from command line was also unsuccessful because there are too many incompatible parts found.

[Mark_Zheng]

Hi Maluber71,

Thanks for your reply. I have tried your solutions but… I have also encoutered multiple issue.
!
Looks like I can not adjust the priority because of denied permission.
And I stuck at downloading tensorflow as well.

[Mark_Zheng]

Hi Weikert,

I agree with you. Somehow the new environment is still set at Python 3.6.13 (perhpas it is because I don’t know how to change that). Many incompatibility issues emerge because of the out-of-date version of python.

[Mark_Zheng]

Dear Marcel IW,

A good news. I tried to install all packages in base, which is my root environment, not the newly created one. After I have installed all packages, a new problem emerged. When I run Bert Model Selector

It shows
ERROR BERT Model Selector 0:171 Execute failed: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
Traceback (most recent call last):
File “”, line 2, in

• File “C:\Users\zheng\Downloads\knime_4.3.3.win32.win32.x86_64\knime_4.3.3\plugins\se.redfield.bert_0.0.1.v202012081121\py\bert_utils.py”, line 8, in load_bert_layer*
• return hub.KerasLayer(bert_model_handle, trainable=True)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\keras_layer.py”, line 146, in init*
• self._func = load_module(handle, tags)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\keras_layer.py”, line 398, in load_module*
• return module_v2.load(handle, tags=tags)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\module_v2.py”, line 97, in load*
• module_path = resolve(handle)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\module_v2.py”, line 53, in resolve*
• return registry.resolver(handle)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\registry.py”, line 42, in call*
• return impl(*args, *kwargs)
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\compressed_module_resolver.py”, line 87, in call*
• return resolver.atomic_download(handle, download, module_dir,*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\resolver.py”, line 415, in atomic_download*
• download_fn(handle, tmp_dir)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\compressed_module_resolver.py”, line 83, in download*
• response = self._call_urlopen(request)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\compressed_module_resolver.py”, line 96, in _call_urlopen*
• return url.urlopen(request)*
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 222, in urlopen*
• return opener.open(url, data, timeout)*
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 525, in open*
• response = self._open(req, data)*
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 542, in _open*
• result = self._call_chain(self.handle_open, protocol, protocol +*
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 502, in _call_chain*
• result = func(args)
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 1397, in https_open*
• return self.do_open(http.client.HTTPSConnection, req,*
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 1357, in do_open*
• raise URLError(err)*
  urllib.error.URLError: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>

Is it another connection issue? Or is it because I have not fix the environment issue yet? Please advise. My greatest thanks.

[Mark_Zheng]

Dear MarcelW,

In conclusion, now the situation as follows: it seems that I can run Bert Model Selector without issues related to packages or environment. But, as I noted in last post, when I choose Settings–TensorflowHub, I will encouter urlopen error[WinError 10060]. When I choose Settings–HuggingFace, I will be stuck in endless processing. That’s basically what I have right now.

Thanks for your time.

Best,
Mark

[MarcelW]

Hi @Mark_Zheng,

Glad to hear that you managed to solve the environment problems! I think the error you still experience is indeed rather a connection issue. Do you have any special proxy or firewall settings in place?
I am not so sure about the problem with HuggingFace, perhaps also a connection issue where the connection attempt does not time out but blocks indefinitely?

Perhaps @Redfield already have experience with this kind of error and can help us out .

Marcel

[Mark_Zheng]

Dear MarceIW,

I have checked my internet setting in KNIME and IE and removed all the proxy setting, however, the issue still persists. I hope @Redfield can help me with that. Thank you for your coordination.

Best,
Mark

[Daniel_Weikert]

Thanks Marcel, appreciate your response and clarification here.
br

[Daniel_Weikert]

Do you really need KNIME for your Bert Model?
Any reason for this?
br

[Mark_Zheng]

Hi Weikert,

Frankly speaking, I don’t know much about coding. That’s why I use KNIME.

Best,
Mark

[Redfield]

Hello @Mark_Zheng

Could you please make screenshots of the Model Selector settings, so I will try to reproduce your problem.

I have a hypothesis what is going on there — the model URLs both for HuggingFace and TensorFlowHub might be changing, so we cannot always track them. One thing I can recommend you to do is to activate the mode to use Remote and Local paths in Advanced tab in Model Selector node. Then you need to download the model from HuggingFace or TensorFlowHub manually and provide a path in your local file system. However there also might be some other problems, since there is no standard for BERT models, and some models will be incompatible with our nodes.

Best regards,
Artem.

[Mark_Zheng]

Dear Artem,

Thanks for the help! I agree with your hypothesis. This is the model selector setting of my Bert Model Selector in DPedia_Text Classification.

And that’s what I saw in the console:
ERROR BERT Model Selector 4:15 Execute failed: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
Traceback (most recent call last):
File “”, line 2, in

• File “C:\Users\zheng\Downloads\knime_4.3.3.win32.win32.x86_64\knime_4.3.3\plugins\se.redfield.bert_0.0.1.v202012081121\py\bert_utils.py”, line 8, in load_bert_layer*
• return hub.KerasLayer(bert_model_handle, trainable=True)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\keras_layer.py”, line 146, in init*
• self._func = load_module(handle, tags)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\keras_layer.py”, line 398, in load_module*
• return module_v2.load(handle, tags=tags)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\module_v2.py”, line 97, in load*
• module_path = resolve(handle)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\module_v2.py”, line 53, in resolve*
• return registry.resolver(handle)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\registry.py”, line 42, in call*
• return impl(*args, *kwargs)
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\compressed_module_resolver.py”, line 87, in call*
• return resolver.atomic_download(handle, download, module_dir,*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\resolver.py”, line 415, in atomic_download*
• download_fn(handle, tmp_dir)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\compressed_module_resolver.py”, line 83, in download*
• response = self._call_urlopen(request)*
• File “C:\Users\zheng\anaconda3\lib\site-packages\tensorflow_hub\compressed_module_resolver.py”, line 96, in _call_urlopen*
• return url.urlopen(request)*
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 222, in urlopen*
• return opener.open(url, data, timeout)*
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 525, in open*
• response = self._open(req, data)*
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 542, in _open*
• result = self._call_chain(self.handle_open, protocol, protocol +*
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 502, in _call_chain*
• result = func(args)
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 1397, in https_open*
• return self.do_open(http.client.HTTPSConnection, req,*
• File “C:\Users\zheng\anaconda3\lib\urllib\request.py”, line 1357, in do_open*
• raise URLError(err)*
  urllib.error.URLError: <urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
  That’s what I saw in the error log:
  
  
  
  So I think you are right that URLs might be changing. I also tried the Local Path approach of model selection, but it seems tensorflowhub does not provde pb file, therefore I haven’t found way to use it.

Any suggestions? Thank you!

Mark

[Redfield]

It looks like it just cannot reach the model. However I could successfully install this model with Model Selector. Are you sure you do not have some firewall restrictions, or proxy-server in the middle?

The workaround that I can suggest for you (I tested it), is to download the model from TF Hub with this URL:
https://tfhub.dev/tensorflow/bert_en_wwm_uncased_L-24_H-1024_A-16/2

Then you need to unzip the archive and using Remote and Local paths mode provide a path to the folder where you unzipped the archive. I hope it will help you.

If not, could you please run ‘pip freeze’ to get the list of packages for Python deep learning environment that you use and post it here?