Google Trends

KNIME Analytics Platform
1

Dear Community.
Did somebody succeed in setting up a workflow that is able to crawl historical data trending searches like this?
https://trends.google.com/trends/explore?cat=3&date=2019-10-01%202019-10-31&geo=DE
I always get a 429 even with the first request. According to my searches the user has to be logged to do a successful request. According to what I saw the token needs to be passed around for successful requests as well. Has anybody solved this using KNIME?
Thx a ton!

2

I am afraid to say there is no API provided by Google which could be used in combination with the workflow @oole created in this thread: Google sheets Reader node read automatically a google spreadsheet file from Google Drive?

However I am still experimenting with IAM Service Account Credentials API. Can’t promise anything but IAM authentication might be the last option available.

3

Seems the IAM access only provides access to the Google Cloud Service roles.

Two remaining opportunities I can imagine about that might to lead to a solution are:

  1. Faking a login by manually logging in and copying the cookie information or some sort of dirty hack which would always require manual actions henceforth being less reliably or …

  2. By utilizing Selenium nodes. @matthias.albus, do you have experience with Google auth in combination with Selenium?

Update: A much more elegant way w/o the auth-struggles would be to export certain topics i.e. to Google Sheets which can be easily access via Knime. Though, there is also a web service offering a connector and several Python projects which might pose a starting point.

Kind regards
Mike

1 Like
4

Hi Mike,
thank you very much for your suggestions. I was not able to find a working library that was able to deliver the historical data of the most popular search terms. pytrends or gtrendsR work well when searching for specific keywords but do not provide the “most popular search terms” functionality (or it just works on a yearly basis). I think will try Selenium next, so if anybody has experience using Selenium on Google Trends let me know!
Have a nice evening and again thanks a lot!
Best, Andy

1 Like
5

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.