Hi all,
I am trying to set up an API integration into this Google API - https://developers.google.com/webmaster-tools/search-console-api-original/v3. This one requires POST Request with api key and OAuth2.0 authentication. This has been difficult and after a while I found this post where Mr./Ms. @oole presents the solution.
This was for Google Drive but I tried to adopt this for “webmasters” api instead of “drive” api. I get to this step and I get access_denied error.
Put the client_id into this URL: https://accounts.google.com/o/oauth2/v2/auth?client_id=<Your_Client_ID>&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code &scope=https://www.googleapis.com/auth/drive.metadata.readonly
Copy this URL to the browser of your choice and follow the instructions
Copy the authentication token to you authentication metandode.
From the error message, it seems this API is not yet a verified google API and to get direct access you need to contact the developer. Do you know if this is the case?
In any case, as you mentioned that authenticating with OAuth might be tricky, I can provide to you an alternative way that can help you to authenticate with certain APIs using JWT instead of OAuth. Most of the process is described here:
You need to change the python snippet to add the private key, the private key ID and your service account. The Private Key and the KeyID you can get from the JSON file you download in the service page when creating a key for your service account (for the private key you have to include the whole string). To get the signed JWT, I use a python snippet and a python package called pyJWT (you could do something similar using Java). So you will need this package to run the script. Finally, to do the final adaptation to your API of choice, you have to change the value in ‘aud’: ‘https://translate.googleapis.com/’ to the correct API.
However, this would only work with APIs that have their definitions published here: https://github.com/googleapis/googleapis So this might not work with the webmasters api.
Wow, @ana_ved, this is so so helpful, thank you very much for taking the time to walk through. In terms of contacting the developer, I think I (somehow) assumed that that’s me, my app that I was creating OAuth2.0 token for in the GCP, but I will review the documentation again and see if I can understand it better.
And I am going to follow your steps to see if I can authenticate with Google Search Console or Webmaster API for POST Requests. Thank you so much again for your kindness and support, @ana_ved!!
@ana_ved- Thanks for this, been trying to do JWT for adwords using java as target users for the component i am building may not have python env. Any guidance on how toe get jwt workign with java?
Hi @ana_ved, again thank you for your support and info. I’m trying chipping away at this effort over the weekend…
Puzzling I am getting this error while using your Python Source (Add private key). I followed the instruction to set up Python environment and I think I have the right info entered, private key, keyID, service account email and ‘aud’: ‘https://www.googleapis.com/webmasters/v3’ which I assume is correct. For the private key, i’m including key = "-----BEGIN PRIVATE KEY-----\nMII…and the end portion. When I execute script, I get the error that jwt is not found. In my local environment, using command line, I installed pyjwt. I’ve been googling to see if I can find any posts or forums about this. It takes time, but feels great making progress and when something works as intended. Thank you!!!
c:\Python38>pip install pyjwt
Requirement already satisfied: pyjwt in c:\python38\lib\site-packages (2.0.1)
No module named ‘jwt’
Traceback (most recent call last):
File “”, line 3, in
ModuleNotFoundError: No module named ‘jwt’
This is unfortunate, from the threads you posted, it seems to be something related to the new update of pyjwt. Can you try reinstalling an older version of pyjwt? Perhaps with pip install pyjwt==1.7.1 as an example.
@Gavin_Attard Hi Gavin, hopefully this will be helpful for you too. I wanted to come up with a Java version example to this, but my Java skills are not the greatest. If you know Java, in the google link I pasted here there is an example how to do this in Java.
Hello @ana_ved,
Now, I’m getting this error with 1.7.1.
Could not deserialize key data.
Traceback (most recent call last):
File “”, line 17, in
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\jwt\api_jwt.py”, line 64, in encode
return super(PyJWT, self).encode(
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\jwt\api_jws.py”, line 113, in encode
key = alg_obj.prepare_key(key)
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\jwt\algorithms.py”, line 207, in prepare_key
key = load_pem_public_key(key, backend=default_backend())
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\cryptography\hazmat\primitives\serialization\base.py”, line 20, in load_pem_public_key
return backend.load_pem_public_key(data)
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py”, line 1124, in load_pem_public_key
self._handle_key_loading_error()
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py”, line 1382, in _handle_key_loading_error
raise ValueError(“Could not deserialize key data.”)
ValueError: Could not deserialize key data.
@alabamian2 There might be something in an upcoming Palladian release which could help you with that. It’s not quite ready yet, but if you want to give it a try (and give feedback!), get in touch at: mail@palladian.ai
Hello @qqilihq, thank you again for your help. I’ll report back with how the Palladian test goes. Here is a quick summary of progress:
I’m attempting to use POST Requests with some Google API services and trying to authenticate with OAuth2.0 or some other means required by Google, namely I am trying to use this API which required POST requests (vs. just GET) Search Console API | Google Developers.
#1
I have tried the method in my original first post and I was actually able to resolve that error and was able to move onto next steps. Right now, I can run first 2 nodes but the 2nd POST Request node does not return Json and the 3rd node is not able to run. I updated the URL from https://www.googleapis.com/oauth2/v4/token to others like https://accounts.google.com/o/oauth2/v2/auth but no success.
#2
I tried what @ana_ved suggested and as above, there is an error with pyjwt.
#3
There are Python script implementation that many others have written and shared. They work as Python and I’m exploring to see how easy/difficult it is to replicate Python steps in KNIME. I don’t know Python so we’ll see. Same goes with Java, there are many example codes and process we may be able to copy over into KNIME?
@alabamian2 I’ll continue our conversation here, now that Palladian 2.4 is officially released. Below you’ll find an example workflow which demonstrates accessing the Google API. It consists of two branches:
the simple, “Hello world” example, which directly uses the acquired access token to access the API. The refresh token will expire after some time (I think it was two hours), so you’ll have to re-authenticate, OR use …:
the better, real-world example, where I use the refresh token (which does not expire) first to get get an access token. This way you can keep the OAuth Connector node executed, and just have to run the workflow starting at the “Get Access Token” workflow.
Hi @qqilihq,
Thank you so much for providing the workflow with refresh token integration. My test is going very well and I can see that the token is persisting over time. I’ll test it some more to see how long it persists. I was going to put this as a batch job on Windows Scheduler so the entire workflow may have to reset and execute, which may require some manual interaction but even in that case I can manually run it and loop through data sets so that I only have to manually start it once every so often. I’ll report back with updates later.
Thank you again @qqilihq for the clever solution!!
