python problems with input column values, missing values

Hello all,
I am using workflow to read keywords from googletrends.
WF_py_gtrends

Works fine so far with existing keywords, but if a keyword does not exist at google trends, then no table is created for me anymore.
Table Creator_with_fail

What can I do? Can I set this under options? > I have already tried, but without success.
Py_options

Or must that be adjusted in the script in python?

Demo_py_gtrends.knwf (13.1 KB)

Hope you can help me!
Greetings
Bernd

1 Like

@sabsab you could try and use a try-catch setting to make the workflow run thru

Depending on what you want to do you might have to provide an empty table with the same structure as the one you are trying to collect at the end of the loop.

I will see if I can have a look at your example later.

2 Likes

Thank you, that would be nice. With try-catch I somehow do not get further :frowning:

1 Like

@sabsab OK it took a minute longer. Maybe there is a more elegant solution but it should work if there is always the same number of Rows with the identical RowID which I understand is the case. If not the try-catch might have to look different.

2 Likes

I think @mlauber71 provided you a nice solution including the workflow.
Another option could be the try - except blocks in python directly
br

2 Likes

unfortunately i am a complete beginner with python. is it possible that you can help me a little?
many grettings, Bernd

currently I have problems with the python packages, can’t test it yet. But I am working on it.

py_prob

thanks a lot for the solution!

1 Like

I have tried it with this code but unfortunately it does not work…

# -*- encoding: cp1252 -*-
# -*- coding: utf-8 -*-
from pytrends.request import TrendReq
pytrend = TrendReq()
kw_list = [flow_variables['column1']]
try:
	pytrend.build_payload(kw_list, timeframe='2018-07-25 2021-07-25', cat=0, geo='DE')
	interest_over_time_df = pytrend.interest_over_time()
except:
	for e in kw_list:
		print(e, "#nn")
output_table = interest_over_time_df
1 Like

@sabsab in the example there is a conda environment propagation that should bring all the necessary packages together in an Anaconda/Miniconda environment. In the annotations of the node there is the code for the YAML file used.

Other that that here are other environments you might try (also some especially for Windows). If you have eg. Miniconda set up in KNIME you should be able just to use the Propagation.

2 Likes

@mlauber71 now i have all extensions installed and Python/Minconda is up to date.

But I can not get it to run.

what’s going wrong?

many greetings
Bernd

1 Like

@sabsab I have updated the workflow to contain a Windows and a Mac environment propagation. You can try and force the creation of a environment.

You could also just try and install and environment with the “py37_knime_2022.yaml” YAML file that is in the “/data/” folder of the workflow from your Miniconda Prompt:

image

conda env create -f="C:\Users\user1234567\knime-workspace\kn_forum_38842_py_gtrends_try_catch\data\py37_knime_2022.yaml"

(you would have to change the path obviously …)

This would then check the exact packages suitable for your environment while resprecting the boundaries set in the file:

name: py37_knime            # Name of the created environment
channels:                   # Repositories to search for packages
- conda-forge
dependencies:               # List of packages that should be installed
- python>=3.7,<3.8          # Python
- py4j                      # used for KNIME <-> Python communication
- nomkl                     # Prevents the use of Intel's MKL
- pandas                    # Table data structures
- jedi                      # Python script autocompletion
- python-dateutil           # Date and Time utilities
- numpy                     # N-dimensional arrays
- cairo                     # SVG support
- pillow                    # Image inputs/outputs
- pyarrow=6.0               # Arrow serialization
- IPython                   # Notebook support
- nbformat                  # Notebook support
- scipy                     # Notebook support
- jpype1                    # Databases
- h5py<3.0 # must be < 3.0 because they changed whether str or byte is returned
- libiconv                  # MDF Reader node
- asammdf=5.19.14           # MDF Reader node
# ----------------------------------------------------
- openpyxl              # Excel Reader
- scikit-learn          # Machine Learning
# Visualization
- matplotlib
- bokeh
- seaborn
# Additional packages
- pytrends

I know that conda environment management is initially not so easy, but once you get it it is not that complicated and you gain much flexibility with regards to your settings.

2 Likes

PUHHH, now it runs!

A bit different than described, but it works for me now.

What did I do:

  • “conda install -c conda-forge pytrends” executed
  • “C:\Users\user123\knime-workspace\kn_forum_38842_py_gtrends_try_catch\data\py37_knime.yml” executed > I didn’t find “py37_knime_2022” and downloaded “py37_knime.yml” here “Bitbucket”.
  • under “Executable Selection” in the Python node set the value “Use KNIME Prefs” (look wf)

Here my prefs for python:

And here the wf:
kn_forum_38842_py_gtrends_try_catch2.knwf (78.2 KB)

Many, many thanks to mlauber71!
Cheers, Bernd

ps: Had hoped that it will not be quite so complicated. Maybe a simple adjustment of the python code is enough :slight_smile:

2 Likes

@sabsab glad it does work now :slight_smile:

OK that would be a step once you have your environment and you have activated the specific one you want to use.

conda activate py37_knime

Well the code was right there in the entry :slight_smile: - the file is in a subfolder /data/ of the workflow which is not visible in the KNIME explorer. You would have to check your file. But you managed to download a YAML file from KNIME’s collection.

The main change I did was limit the channel to “conda-forge” which is because of Anaconda changing its license conditions, so in order to stay ‘free’ you might have to limit your usage to this channel (plus some extra things via pip).

4 Likes

So you fixed it. Congratulations.
I was not aware of yaml files availabe for KNIME for download. Interesting. Thanks for sharing

2 Likes

Me neither until recently. What a treat :slight_smile:

2 Likes

Just a small follow up. You are now using the base environment of python which is fine. I just thought you had problems with that and therefore suggested a new environment but now it seems possible the pytrends package was not installed in the first place. That option had not occurred to me - and very possibly was fixed by the conda install …

From my experience if you use python a lot you will have to create and re-install environments, versions and packages all the time …

Hello @mlauber71

Yes, the original problem was to filter out keywords that cannot be processed by Google Trends. This could have been solved directly in the Python code (pytrend) or a Knime solution like this one.
So this is a great solution to work through a list of keywords in Google Trends.
The Python environment was previously at least such that I could run pytrend.

Python and all possible customizations are still rather trial and error for me - but I’m working on it :slight_smile:

Thanks again for the great Knime solution!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.