URGENT ISSUE| GA Authentication | Interactive | quota exceeded

[Gavin_Attard]

Hi good folks of Knime

It appears the API authentication against your cloud console project “868899538500” is now exceeding (straight away at the start of the day) the 50L per day per project quota. This means use of the Google Analytics connector within knime is blocked.

Can you either

Increase the Quota (quickest route to unblock)
Allow a third oAuth option to use our own OAUth credentials and bypass using your project quota.

thanks

Gavin

[hotzm]

Hi @Gavin_Attard,

I cannot look into what is going on with our project at the moment, but in the meantime you could try the Google Authenticator (API Key).
You need to set up your own cloud project and obtain credentials which can then be put into the node config. It is not interactive, but could be a viable option.

The Google Analytics Connector node does not really do much apart from choosing a property under the authenticated account. It will work regardless of which Google Authenticator (Interactive with our project or API key with yours) is in front of it (this is why it is the same port object).

I’ll update when I know more.

[mlauber71]

@Gavin_Attard @hotzm could you elaborate. Why is a node using a knime specific project? I was under the impression one would use their own project (id).

@tobias.koetter can you please weight in or someone with knowledge about these things. This possibly can have huge implications.

[diego_rod_lop]

Hey @Gavin_Attard,

Meanwhile, the answer about the KNIME project arrives. If you want to keep the interactive mode within the Google Authenticator node, you can use your own Google Cloud console project by selecting “Show advanced settings” and then flagging “Use custom client ID.” in the node configuration dialogue .This will “charge” the API requests directly to your project, impacting your own quota.

To do this, you need first:

1. To create an OAuth Client ID (Desktop application) in your Google Cloud Console project and download it.

2. Enable the Google Analytics API, or the API that you need to use, in your Google Cloud project.

Then it is possible to download the JSON file to your local system or put it in some KNIME Hub instance with execution capabilities.

For example if you choose “Local File System” you need to provide the path where the JSON file is saved in your computer, the JSON file looks like this

{"installed":
    {"client_id":"your-client-id",
    "project_id":"your-project-name",
    "auth_uri":"https://accounts.google.com/o/oauth2/auth",
    "token_uri":"https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs",
    "client_secret":"your-client-secret",
    "redirect_uris":["http://localhost"]}
}

Otherwise it is possible also use your Google Cloud project by generating an API key from a Service Account as @hotzm suggested.

I hope this could be helpful somehow.

[Gavin_Attard]

HI All

Thanks for all the input, ti was very helpful.

Here is some feedback on some of the questions and solutions posed:

1. Using API key authenticator via service account
   It is not normally possible for most users of GA to have cloud access to easily create the keys, there can be some significant delay depending on organization and process… In addition, it is worth noting that the service account will also need adding to the respective GA accounts in order to access the data. If you are working across several accounts/businesses this can have its own challenges and almost certainly not quick.

2. Using Google Authenticator with Client ID setting from Advanced Settings
   Thanks for the pointer on this, had missed it. When we tried it out there is a mismatch between the redirect URI setup in the Cloud console consent screen setup and what the tool is setting. This means the option is not usable.
   FIX: please make the redirect_uri field an input field in addition to the JSON field or clearly document what should be the redirect uri (we couldn’t spot any)

3. How we solved this
   We setup our own Oauth ID and setup our own consent screen.
   This is quite easy to do for anyone, as a cloud project can be created with ease.

Create project if you dont’ have one
Add relevant APIs from the Library
Setup Consent screen - Set it it to Internal, or if you have multiple users who need it, use External in Testing mode (gives you upto 100 users you can add).
Create an OAuth Id
We then used the Oauth 2 Authenticator together with a toll we borrowed from another user to enter the client ID and secret (Note to dev - if you can add the JSON option to the OAuth2 Authenticator, you make this process so much easier…)

This worked and created the consent screen to choose the account/id you have the GA accounts associated with and worked a treat

4. Why this is a problem @mlauber71 -
   In order for the Knime Google Authenticator to work in Interactive mode, Knime have a cloud project with the APIs, Oauth CLient id and consent screen setup. When we login and choose our google identity, we are associating that account ti make api calls via the Knime project. Each project has a default 50K calls per day. At present, usage of the tool (presumable cause everyone under the sun is archiving GA3 data or its usage has grown to such), that threshold is reached almost instantly every day.
   The profile selector tool doesn’t work and any queries you try and do wont work.

FInal thoughts.
I expect that an increase in calls per day has a monetary implication, which for an open source product i would not expect Knime to pick up.
I think looking to resolving the Fix highlighted in Item 2 will resolve this easily and with proper documentation (i can help here) user should be able to easily adjust.

Much love

Gavin

[diego_rod_lop]

Hey @Gavin_Attard ,

Thanks for the comprehensive feedback and the heads-up on how you solved it.

So, if understood correctly, regarding point 2, you finally authenticated using the OAuth Authenticator connector connected to the Google Analytics Connector, right?

I agree that adding a bit more context in the Google Authenticator node description doesn’t hurt at all Mainly explaining how to set up the OAuth Consent Screen.

However, I was able to use an OAuth 2.0 Client ID (Desktop application) from a personal Google Cloud project to authenticate via the Google Authenticator node. You were right that it is necessary to configure the OAuth consent screen, too, in my case, adding the scope of the API I want to use.

To understand your issue a bit more, what kind of OAuth 2.0 Client ID are you using, and which error are you getting from the Google Authenticator node?

Thanks a lot in advance for the information.

[Gavin_Attard]

HI @diego_rod_lop

on point 2 - i used the Google Authenticator with Advanced setting of own client id on.
The error we got was a mismatch between the referral uri in the client id we set in cloud console and the referral id the Google Authenticator has set.
If this is made visible we can set our redirect URI to that, or make it editable to se can enter our own referral uri ( i woudl imagine the former is preferable if you are using some kind of standard localhost variant)

As for the Oauth connector we used here is a screen shot

[andremafei1]

You probably need to contact Google to increase the Knime Google Cloud Project rate limit. My guess it that with the date when Google will delete all GA Universal Analytics historic data getting close (Jul 1, 2024), lot’s of people are using Knime to backup their GA data and it reached the Knime Google Cloud Project limit.

[Gavin_Attard]

@diego_rod_lop - for the Google Authenticator, woudl you know the redirect_url the knime project is using?

I can then match my projects to that and it should work.

Thanks

[Gavin_Attard]

found it: http://localhost:52318

[Gavin_Attard]

bah! no luck, it keeps changing the url expectation for some reason… why would it make one up? humpf.

i even tried having no redirect_url in the console setup and that made no difference.

[diego_rod_lop]

Apologies for the delayed response.

I see that you have already resolved your issue in this thread . Is it correct?

Is there anything else we can help you with?

Best regards

[andremafei1]

Hi @diego_rod_lop, no, this is a different problem that is still happening:

[diego_rod_lop]

Hey @andremafei1,

Thanks for reaching out; your assumption is probably correct: many GA queries use the KNIME default project.

We also need to update the example workflow because there is a newer version of the Google Authenticator node.

Meanwhile we think about how to handle the Google Analytics API Quota limit for the KNIME project; you can use the new Google Authenticator node, with your own Google Cloud project to fetch data from GA. As I mentioned above in this thread .

Let me know if it could help.

[Gavin_Attard]

Ya, the issue is still ongoing, but as i said earlier, the core issue is not the usageLImit, for a opensoruce free tool, i don’t expect you guyts to up the limit.

However what we need is a way to use our own oAuth credentials (gcp project) to connect to google.

If i try to do that using the Google AUth tool, i get a redirect url mismatch error. When i thought i figured out what the redirect_url should be, it went and switched it in teh Google Auth node… SO i can never get it to work.

So instead i use the oAuth2 Connector, That works as i can set the redirect_url to whatever i want, however the oAUth2 Connector either seems to lose the token, or it assumes it expires after a certain period, either way, it loses the token, and there is no way to get it to seek a refresh.

[diego_rod_lop]

Ahh, @Gavin_Attard, I see your point; you have already explained it here.

I’m a bit confused about why you’re receiving this error . Suppose you’re trying to retrieve Google Analytics data using a local workflow in the KNIME Analytics Platform. In that case, you must generate a Desktop app type OAuth Client ID in your Google Cloud Project. This way, you won’t need to provide any redirect URI.

What kind of OAuth Client ID are you using to authenticate towards the Google Analytics API?

BR

[andremafei1]

I think it is really important to fix this because I guess many people use Knime to download data from GA because they are not comfortable or can’t use Google Cloud due to their companies’ internal limitations/reasons (security, knowledge, politics, etc).

[Gavin_Attard]

We are using Web Application. I did not know that desktop app does not create a redirect url requirement.

I will try that. THanks

[Gavin_Attard]

The Definitive Guide to getting round this and setting up your own OAuth Client ID

Ok folks, so lets recap on the problem

1. The Knime GCP project is knackered as everyone under the sun is doing loads of queries per day so hitting easily to 50K limit.

Solution:
You need to create your own oAuth Consent screen and client ID to give yourself your very own 50K per day limit.

1. Create a GCP project
2. Go to APIs and add the Google Analytics API
3. Configure the oAuth Consent Screen
4. Create an OAuth client id as a desktop app
5. Download the JSON and use that to login with the Google Authenticator node.
   
   
   
   
   
   
   
   
   
   
   
   

Happy GA3 Archiving.
note - this is also valid for use with GA4 Data api. you just need to add the Api rom the library in gcp.

[andremafei1]

I consider this a good workaround the problem, not the solution of root cause of the problem.