ModuleNotFoundError: No module named 'google.cloud'

[anguslou]

I have the same codes successfully run on Jupyter notebook but not in Knime Python Script Node, both of them are running in the same conda virtual environment, and I connect the Knime Python Script with Conda Environment Propagation Node (to the same virtual environment I am running on Jupyter Notebook).

However, in Knime Python Script, I get an error when running the following line:

from google.cloud import vision

ModuleNotFoundError: No module named ‘google.cloud’

I am running on Mac M1 chip but it should not be the reason if I can run it on Jupyter notebook.

Is it possible to run Jupyter notebook directly in Knime instead of the Python Script node?

[anguslou]

Thinking twice, the input variables are in Knime so running Jupyter in Knime may not work unless I can feed in variables to the Jupyter notebook running in Knime.

[mlauber71]

@anguslou it is possible to transfer the content of Flow Variables to and from Jupyter notebooks from within KNIME. It is not very elegant though:

UnicodeEncodeError: 'charmap' codec can't encode characters

OK I have added a part to the workflow where KNIME, Python and Jupyter would communicate with each other. ‘Variables’ can be exchanged via a file and also data can be transferred via Parquet files. It would be great though to be able to transfer ‘real’ variables between KNIME and Jupyter (like in this idea). [image] Unfortunately on my Mac the export thru the new Python integration produces a strange error about index when trying to access the results. Some nodes (like Group by) although wou…

Please also note the option to only execute parts of the Jupyter notebook (only tagged fields) in order to make the cooperation more comfortable.

Use functions and Python code from an external .PY file or a Juypter notebook...

Though it might be that this is not the problem in your case. Would it be possible to provide a minimal example or a log in debug mode.

[anguslou]

I can share the log in google drive, too big to upload here.

knime log.txt

[mlauber71]

@anguslou these things. I would be better to clear the log of older entries and focus on the problem. Then the error message says the the Python environment you are using does not have the necessary package.

First you should make sure that your Python node indeed does use the python environment (“threeKingdoms”, “kn_3kingdoms” ?) you want to work with.

Then it seems that you might have to explicitly install specific versions of google cloud python packages (and maybe call them)

No module named 'google.cloud' in Python | bobbyhadz

Also there might be additional communication settings be necessary between Python and google cloud.

Best would be if you could provide us with a minimal example that would demonstrate the problem or maybe a set of the code.

[anguslou]

That is one of the questions: I run the same code under the same virtual environment, but it works in Jupyter notebook but not in Knime.

[mlauber71]

@anguslou maybe you could provide screenshots of your configuration settings for Conda and the environments in the settings and with the Python node.

[anguslou]

Sorry for the inconvenience, i repost the log here

2022-10-23 19:22:48,301 : INFO : KNIME-Worker-13-Conda Environment Propagation 3:8 : : CondaEnvironmentPropagationNodeModel : Conda Environment Propagation : 3:8 : Environment ‘3kingdom’ does not exist. Creating the environment. This might take a while…
2022-10-23 19:23:34,969 : WARN : Thread-22 : : CondaEnvironmentPropagationNodeModel : Conda Environment Propagation : 3:8 : ==> WARNING: A newer version of conda exists. <==
2022-10-23 19:23:34,976 : WARN : Thread-22 : : CondaEnvironmentPropagationNodeModel : Conda Environment Propagation : 3:8 : current version: 4.12.0
2022-10-23 19:23:34,976 : WARN : Thread-22 : : CondaEnvironmentPropagationNodeModel : Conda Environment Propagation : 3:8 : latest version: 22.9.0
2022-10-23 19:23:34,976 : WARN : Thread-22 : : CondaEnvironmentPropagationNodeModel : Conda Environment Propagation : 3:8 : Please update conda by running
2022-10-23 19:23:34,976 : WARN : Thread-22 : : CondaEnvironmentPropagationNodeModel : Conda Environment Propagation : 3:8 : $ conda update -n base -c defaults conda
2022-10-23 19:24:12,191 : INFO : KNIME-Worker-13-Conda Environment Propagation 3:8 : : LocalNodeExecutionJob : Conda Environment Propagation : 3:8 : Conda Environment Propagation 3:8 End execute (1 min, 24 secs)
2022-10-23 19:24:12,194 : DEBUG : KNIME-Worker-13-Conda Environment Propagation 3:8 : : WorkflowManager : Conda Environment Propagation : 3:8 : Conda Environment Propagation 3:8 doBeforePostExecution
2022-10-23 19:24:12,196 : DEBUG : KNIME-Worker-13-Conda Environment Propagation 3:8 : : NodeContainer : Conda Environment Propagation : 3:8 : Conda Environment Propagation 3:8 has new state: POSTEXECUTE
2022-10-23 19:24:12,197 : DEBUG : KNIME-Worker-13-Conda Environment Propagation 3:8 : : WorkflowManager : Conda Environment Propagation : 3:8 : Conda Environment Propagation 3:8 doAfterExecute - success
2022-10-23 19:24:12,200 : DEBUG : KNIME-Worker-13-Conda Environment Propagation 3:8 : : NodeContainer : Conda Environment Propagation : 3:8 : Conda Environment Propagation 3:8 has new state: EXECUTED
2022-10-23 19:24:12,221 : DEBUG : KNIME-Worker-13-Conda Environment Propagation 3:8 : : Node : Python Script : 3:4 : Configure succeeded. (Python Script)
2022-10-23 19:24:12,222 : DEBUG : KNIME-Worker-13-Conda Environment Propagation 3:8 : : NodeContainer : Conda Environment Propagation : 3:8 : Python Script 3:4 has new state: CONFIGURED_MARKEDFOREXEC
2022-10-23 19:24:12,223 : DEBUG : KNIME-Worker-13-Conda Environment Propagation 3:8 : : NodeContainer : Conda Environment Propagation : 3:8 : Python Script 3:4 has new state: CONFIGURED_QUEUED
2022-10-23 19:24:12,227 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : WorkflowManager : Python Script : 3:4 : Python Script 3:4 doBeforePreExecution
2022-10-23 19:24:12,227 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : NodeContainer : Python Script : 3:4 : Python Script 3:4 has new state: PREEXECUTE
2022-10-23 19:24:12,229 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : WorkflowDataRepository : Python Script : 3:4 : Adding handler dd3c678b-0be0-4364-84c9-fca4c5938958 (Python Script 3:4: ) - 4 in total
2022-10-23 19:24:12,229 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : WorkflowManager : Python Script : 3:4 : Python Script 3:4 doBeforeExecution
2022-10-23 19:24:12,230 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : NodeContainer : Python Script : 3:4 : Python Script 3:4 has new state: EXECUTING
2022-10-23 19:24:12,230 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : LocalNodeExecutionJob : Python Script : 3:4 : Python Script 3:4 Start execute
2022-10-23 19:24:15,829 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : Python2KernelBackend : Python Script : 3:4 : Python PID: 22127
2022-10-23 19:24:15,834 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : Python2KernelBackend : Python Script : 3:4 : Using serialization library: Flatbuffers Column Serialization.
2022-10-23 19:24:16,379 : DEBUG : python-kernel-creator : : Python2KernelBackend : : : Python PID: 22142
2022-10-23 19:24:17,961 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : Node : Python Script : 3:4 : reset
2022-10-23 19:24:17,961 : ERROR : KNIME-Worker-15-Python Script 3:4 : : Node : Python Script : 3:4 : Execute failed: No module named ‘google.cloud’
Traceback (most recent call last):
File “”, line 13, in
ModuleNotFoundError: No module named ‘google.cloud’

org.knime.python2.kernel.PythonIOException: No module named ‘google.cloud’
Traceback (most recent call last):
File “”, line 13, in
ModuleNotFoundError: No module named ‘google.cloud’

at org.knime.python2.kernel.Python2KernelBackend.executeCommand(Python2KernelBackend.java:1282)
at org.knime.python2.kernel.Python2KernelBackend.execute(Python2KernelBackend.java:1238)
at org.knime.python2.kernel.Python2KernelBackend.lambda$10(Python2KernelBackend.java:1244)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

Caused by: Traceback (most recent call last):
File “/Applications/KNIME 4.5.2.app/Contents/Eclipse/plugins/org.knime.python2_4.6.0.v202206100850/py/messaging/RequestHandlers.py”, line 96, in _handle_custom_message
response = self._respond(message, response_message_id, workspace)
File “/Applications/KNIME 4.5.2.app/Contents/Eclipse/plugins/org.knime.python2_4.6.0.v202206100850/py/messaging/RequestHandlers.py”, line 379, in _respond
output, error = workspace.execute(source_code, request.id)
File “/Applications/KNIME 4.5.2.app/Contents/Eclipse/plugins/org.knime.python2_4.6.0.v202206100850/py/PythonKernelBase.py”, line 328, in execute
exec(source_code, self._exec_env, self._exec_env)
File “”, line 13, in
ModuleNotFoundError: No module named ‘google.cloud’

2022-10-23 19:24:17,973 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : WorkflowManager : Python Script : 3:4 : Python Script 3:4 doBeforePostExecution
2022-10-23 19:24:17,975 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : NodeContainer : Python Script : 3:4 : Python Script 3:4 has new state: POSTEXECUTE
2022-10-23 19:24:17,976 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : WorkflowManager : Python Script : 3:4 : Python Script 3:4 doAfterExecute - failure
2022-10-23 19:24:17,977 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : Node : Python Script : 3:4 : reset
2022-10-23 19:24:17,977 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : Node : Python Script : 3:4 : clean output ports.
2022-10-23 19:24:17,978 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : WorkflowDataRepository : Python Script : 3:4 : Removing handler dd3c678b-0be0-4364-84c9-fca4c5938958 (Python Script 3:4: ) - 3 remaining
2022-10-23 19:24:17,978 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : NodeContainer : Python Script : 3:4 : Python Script 3:4 has new state: IDLE
2022-10-23 19:24:17,979 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : Node : Python Script : 3:4 : Configure succeeded. (Python Script)
2022-10-23 19:24:17,979 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : NodeContainer : Python Script : 3:4 : Python Script 3:4 has new state: CONFIGURED
2022-10-23 19:24:18,005 : DEBUG : KNIME-Worker-15-Python Script 3:4 : : NodeContainer : Python Script : 3:4 : 3k_extract from image 3 has new state: CONFIGURED
2022-10-23 19:24:43,492 : DEBUG : main : : WorkflowEditPartFactory : : : Deactivate org.knime.workbench.editor.context

[mlauber71]

@anguslou it seems your environment ‘3kingdom’ is not there at the start and is being created in the process. You might want to create it first and then try again. Also I would recommend not using an integer at the start of the environment name.

Then could you check if the usage of the mentioned environment is configured at the Python Script node like this:

Next step might be to create a new environment from a fresh YAML file and try to add the google packages exclusively from conda-forge

Python Environment Creation Problem

@kienerj on a Windows machine you can (currently) see the YAML files used under: \knime_4.5.x\plugins\org.knime.python2.envconfigs_4.5.x.yyyyyyyyyy\envconfigs that can help to build one’s own configuration. If you already have built an environment and want to save it in a ‘clean’ (reusable) way the way to go seems to be the --from-history function: Linux … conda env export --from-history | grep -v “prefix” > environment.yaml Windows: conda env export --from-history | findstr -v “pref…

Which Python versions works better with Knime?

@JayR the thing is you will have to find the right combination of Python version and packages that would work with KNIME as well as your individual packages What I would do install Miniconda on your system (how to manage environments) Choose a major Python version (like 3.8) that would most likely work with you package look up a suitable YML file provided by KNIME (python version and operating system) edit the file so it would suite your initial needs. Use only ‘conda-forge’ and add additiona…

[anguslou]

I make a new conda env without an integer at the start, and connects to a new Python script node, it works now.

Thank you.