Having difficulties setting up Post Request to Google API requiring OAuth2.0

Hi all,
I am trying to set up an API integration into this Google API - https://developers.google.com/webmaster-tools/search-console-api-original/v3. This one requires POST Request with api key and OAuth2.0 authentication. This has been difficult and after a while I found this post where Mr./Ms. @oole presents the solution.


This was for Google Drive but I tried to adopt this for “webmasters” api instead of “drive” api. I get to this step and I get access_denied error.

  1. Put the client_id into this URL:
    https://accounts.google.com/o/oauth2/v2/auth?client_id=<Your_Client_ID>&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code &scope=https://www.googleapis.com/auth/drive.metadata.readonly
  2. Copy this URL to the browser of your choice and follow the instructions
  3. Copy the authentication token to you authentication metandode.

I suppose this isn’t KINME problem so I’ll google some more in non-KNIME specific way. But, I’ll want to build the workflow in KNIME.

Thank you so much in advance for your help!!!

Hi @alabamian2

From the error message, it seems this API is not yet a verified google API and to get direct access you need to contact the developer. Do you know if this is the case?

In any case, as you mentioned that authenticating with OAuth might be tricky, I can provide to you an alternative way that can help you to authenticate with certain APIs using JWT instead of OAuth. Most of the process is described here:

I have an example workflow here: https://hub.knime.com/ana_ved/spaces/Public/latest/Translate%20annonimized . This was a workflow I created to access translate API with a service account, for instance.

You need to change the python snippet to add the private key, the private key ID and your service account. The Private Key and the KeyID you can get from the JSON file you download in the service page when creating a key for your service account (for the private key you have to include the whole string). To get the signed JWT, I use a python snippet and a python package called pyJWT (you could do something similar using Java). So you will need this package to run the script. Finally, to do the final adaptation to your API of choice, you have to change the value in ‘aud’: ‘https://translate.googleapis.com/’ to the correct API.

However, this would only work with APIs that have their definitions published here: https://github.com/googleapis/googleapis So this might not work with the webmasters api.

4 Likes

Wow, @ana_ved, this is so so helpful, thank you very much for taking the time to walk through. In terms of contacting the developer, I think I (somehow) assumed that that’s me, my app that I was creating OAuth2.0 token for in the GCP, but I will review the documentation again and see if I can understand it better.

And I am going to follow your steps to see if I can authenticate with Google Search Console or Webmaster API for POST Requests. Thank you so much again for your kindness and support, @ana_ved!!

2 Likes

@ana_ved- Thanks for this, been trying to do JWT for adwords using java as target users for the component i am building may not have python env. Any guidance on how toe get jwt workign with java?

kr

1 Like

Hi @ana_ved, again thank you for your support and info. I’m trying chipping away at this effort over the weekend…

Puzzling I am getting this error while using your Python Source (Add private key). I followed the instruction to set up Python environment and I think I have the right info entered, private key, keyID, service account email and ‘aud’: ‘https://www.googleapis.com/webmasters/v3’ which I assume is correct. For the private key, i’m including key = "-----BEGIN PRIVATE KEY-----\nMII…and the end portion. When I execute script, I get the error that jwt is not found. In my local environment, using command line, I installed pyjwt. I’ve been googling to see if I can find any posts or forums about this. It takes time, but feels great making progress and when something works as intended. Thank you!!! :slight_smile:

c:\Python38>pip install pyjwt
Requirement already satisfied: pyjwt in c:\python38\lib\site-packages (2.0.1)

No module named ‘jwt’
Traceback (most recent call last):
File “”, line 3, in
ModuleNotFoundError: No module named ‘jwt’

duh, I needed to install in conda environment. I’m now getting different errors. :slight_smile:

hhmm, googling some more.

load_pem_private_key() missing 1 required positional argument: ‘backend’
Traceback (most recent call last):
File “”, line 17, in
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\jwt\api_jwt.py”, line 63, in encode
return api_jws.encode(json_payload, key, algorithm, headers, json_encoder)
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\jwt\api_jws.py”, line 110, in encode
key = alg_obj.prepare_key(key)
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\jwt\algorithms.py”, line 250, in prepare_key
key = load_pem_private_key(key, password=None)
TypeError: load_pem_private_key() missing 1 required positional argument: ‘backend’

Is this an ongoing issue with PYJWT?

This was updated on Oct 27, 2020 so it must have worked just a few months ago. Researching some more.

Hi @alabamian2

This is unfortunate, from the threads you posted, it seems to be something related to the new update of pyjwt. Can you try reinstalling an older version of pyjwt? Perhaps with pip install pyjwt==1.7.1 as an example.

@Gavin_Attard Hi Gavin, hopefully this will be helpful for you too. I wanted to come up with a Java version example to this, but my Java skills are not the greatest. If you know Java, in the google link I pasted here there is an example how to do this in Java.

1 Like

Hello @ana_ved,
Now, I’m getting this error with 1.7.1.

Could not deserialize key data.
Traceback (most recent call last):
File “”, line 17, in
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\jwt\api_jwt.py”, line 64, in encode
return super(PyJWT, self).encode(
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\jwt\api_jws.py”, line 113, in encode
key = alg_obj.prepare_key(key)
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\jwt\algorithms.py”, line 207, in prepare_key
key = load_pem_public_key(key, backend=default_backend())
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\cryptography\hazmat\primitives\serialization\base.py”, line 20, in load_pem_public_key
return backend.load_pem_public_key(data)
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py”, line 1124, in load_pem_public_key
self._handle_key_loading_error()
File “C:\Users\Tyson.Nakayama\Anaconda3\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py”, line 1382, in _handle_key_loading_error
raise ValueError(“Could not deserialize key data.”)
ValueError: Could not deserialize key data.

@alabamian2 There might be something in an upcoming Palladian release which could help you with that. It’s not quite ready yet, but if you want to give it a try (and give feedback!), get in touch at: mail@palladian.ai

–Philipp

1 Like

Hi @qqilihq, thank you for reaching out. Sure, I will send an email request and try it out their new feature. Thank you for your suggestion!!

1 Like

Hello @qqilihq, thank you again for your help. I’ll report back with how the Palladian test goes. Here is a quick summary of progress:

I’m attempting to use POST Requests with some Google API services and trying to authenticate with OAuth2.0 or some other means required by Google, namely I am trying to use this API which required POST requests (vs. just GET) Search Console API  |  Google Developers.

#1
I have tried the method in my original first post and I was actually able to resolve that error and was able to move onto next steps. Right now, I can run first 2 nodes but the 2nd POST Request node does not return Json and the 3rd node is not able to run. I updated the URL from https://www.googleapis.com/oauth2/v4/token to others like https://accounts.google.com/o/oauth2/v2/auth but no success.

image

#2
I tried what @ana_ved suggested and as above, there is an error with pyjwt.

#3
There are Python script implementation that many others have written and shared. They work as Python and I’m exploring to see how easy/difficult it is to replicate Python steps in KNIME. I don’t know Python so we’ll see. Same goes with Java, there are many example codes and process we may be able to copy over into KNIME?

Thank you so much for your help, @qqilihq!!

1 Like

@alabamian2 I’ll continue our conversation here, now that Palladian 2.4 is officially released. Below you’ll find an example workflow which demonstrates accessing the Google API. It consists of two branches:

  • the simple, “Hello world” example, which directly uses the acquired access token to access the API. The refresh token will expire after some time (I think it was two hours), so you’ll have to re-authenticate, OR use …:

  • the better, real-world example, where I use the refresh token (which does not expire) first to get get an access token. This way you can keep the OAuth Connector node executed, and just have to run the workflow starting at the “Get Access Token” workflow.

Find the example workflow on my NodePit Space:

Hope this helps. Any feedback regarding the new node is very welcome!

– Philipp

4 Likes

Hi @qqilihq,
Thank you so much for providing the workflow with refresh token integration. My test is going very well and I can see that the token is persisting over time. I’ll test it some more to see how long it persists. I was going to put this as a batch job on Windows Scheduler so the entire workflow may have to reset and execute, which may require some manual interaction but even in that case I can manually run it and loop through data sets so that I only have to manually start it once every so often. I’ll report back with updates later.

Thank you again @qqilihq for the clever solution!!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.