Performance issues when transferring data: Google BigQuery to Knime

Hello,

I have a question regarding the performance of transferring data via connectors. I am using Knime version 4.7.6. on a standard notebook.

Via the nodes Google Authenfification (API Key), Google BiqQuery Connector, DB Table Selector and DB Reader (see graphic) I can access tables in Google BigQuery. This works without errors for small tables (e.g. 3000 records, few columns).

However, if I want to access a larger table (e.g. one million records, few columns) in Google BigQuery in the same way, this unfortunately does not work. I then receive no error message and the DB Reader works for hours and does not finish processing, so I have to abort at some point.

Are there any tips on how I can fix this problem? Is it possible, for example, to speed up the process by changing some settings? Or is - with a normal notebook - the data transfer via the Google BiqQuery Connector always comparatively slow and fails with large amounts of data?

Hi @bekop_veo -

I’m guessing here, but this sounds like it might be a rate limit problem. Is it possible that your administrator has set a limit on the amount of data you can access at once? Are you able to access this data using a tool other than KNIME?

Dear Scott,

Thank you very much for your feedback.

Due to a vacation, I have three weeks no access to the database environment where the described problem occurred. I will check after my return if your hint solves the problem.

Thanks again

Hello @bekop_veo ,
can you please update to the latest JDBC driver for BigQuery as described here. KNIME Analytics Platform shouldn’t be the problem since the whole query processing happens within BigQuery and KNIME only uses the JDBC driver to retrieve the result. Once the query is processed and the first result rows are returned you should also see it via the tooltip and progress change of the DB Reader node. So if this doesn’t happen within a reasonable time there might be a problem with your query or the BigQuery setup and you might want to ask Google support or your BigQuery admin regarding any limitations or timeouts as Scott suggested.
Bye
Tobias

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.