KNIME DB Loader Execute failed: Error getting access token for service account: connect timed out

KNIME Server
1

Hi

I’m using Googe Authenticator (API Key) with Google BigQuery. When I execute DB loader. on desktop (windows) it works fine, but when I try to deploy on server (ubunut, AWS AMI image) I get this error message “DB Loader : 0:163 : Execute failed: Error getting access token for service account: connect timed out”.

In the same workflow I also use DB Table Selector, DB Reader, DB Writer works and they seem to work okay. I create a dedicated Google Authentication (key) + BigQuery connector for DB Loader it still have the same issue when I deploy to the server.

Full error message below, any ideas?

2021-09-16 15:22:05,585 : ERROR : KNIME-Worker-319-DB Loader 0:163 : 05e1beef-49e7-4468-816a-49c50f3c51a7 : Node : DB Loader : 0:163 : Execute failed: Error getting access token for service account: connect timed out
com.google.cloud.bigquery.BigQueryException: Error getting access token for service account: connect timed out
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:106)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.open(HttpBigQueryRpc.java:602)
at com.google.cloud.bigquery.TableDataWriteChannel$2.call(TableDataWriteChannel.java:87)
at com.google.cloud.bigquery.TableDataWriteChannel$2.call(TableDataWriteChannel.java:82)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.TableDataWriteChannel.open(TableDataWriteChannel.java:81)
at com.google.cloud.bigquery.TableDataWriteChannel.(TableDataWriteChannel.java:41)
at com.google.cloud.bigquery.BigQueryImpl.writer(BigQueryImpl.java:1244)
at org.knime.database.extension.bigquery.agent.BigQueryDBLoader.load(BigQueryDBLoader.java:151)
at org.knime.database.extension.bigquery.node.io.load.BigQueryLoaderNode2.load(BigQueryLoaderNode2.java:326)
at org.knime.database.node.io.load.node.DBLoaderNodeModel2.load(DBLoaderNodeModel2.java:335)
at org.knime.database.node.io.load.node.DBLoaderNodeModel2.execute(DBLoaderNodeModel2.java:239)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:556)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1259)
at org.knime.core.node.Node.execute(Node.java:1039)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:365)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:219)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: java.io.IOException: Error getting access token for service account: connect timed out
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:432)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:157)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:145)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:91)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:88)
at com.google.api.client.http.HttpRequestFactory.buildPostRequest(HttpRequestFactory.java:128)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.open(HttpBigQueryRpc.java:597)
… 25 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsClient.(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsClient.New(Unknown Source)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(Unknown Source)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:108)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:79)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:995)
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:429)
… 32 more

2

Hello Mike,

As I understand it, you are developing a workflow that is deployed to your KNIME Server instance, and when you run it as a job on the server, you get:
DB Loader : 0:163 : Execute failed: Error getting access token for service account: connect timed out

With the full log line of:
2021-09-16 15:22:05,585 : ERROR : KNIME-Worker-319-DB Loader 0:163 : 05e1beef-49e7-4468-816a-49c50f3c51a7 : Node : DB Loader : 0:163 : Execute failed: Error getting access token for service account: connect timed out
com.google.cloud.bigquery.BigQueryException: Error getting access token for service account: connect timed out

If this is correct, please confirm and I will research the issue further.

Please provide the following for additional information/troubleshooting:

  • What operating system is your KNIME software running on?
  • What version of KNIME Server (KS)/KNIME Executor are being used?
  • What version of KNIME Analytics Platform (AP) is being used?
  • Does the issue affect one user, multiple users, or all users?
  • Have there been any external changes to the environment that could explain the issue (e.g. configuration changes, new patch applied, new version of component)
  • Has this workflow worked before when deployed to the server (as in something has changed recently), or has it only worked when run inside of AP?

I look forward to helping you resolve this issue!

Regards,
Nickolaus

3

Hi Nickolaus

Yes, you’re right. In answer to your question:

  • Knime is running on Knime AWS AMI image
  • Running Knime Server 4.13.1 / Knime Executor 4.4.1
  • Knime AP on desktop is 4.41 running on Windows
  • We just started with Knime Server so its affecting one user, but trying different server user account it has the same problem
  • We’ve done an upgrade of the knime server/executor from the base AMI image, installed the BQ ODBC driver is the primary changes that has been made
  • Only got Knime Server recently, all other DB components work, including DB writer. it’s just that DB writer is too slow at the moment because it writes one record at a time

Regards

Mike

4

Hello,

As I understand it, this issue has been resolved through direct support, in which creating a workflow with an empty table load without any transformations and the DB Loader worked; after that, you added another Google Big Query to the end and that resolved the issue.

If this is not correct, and you still need additional assistance, please let us know.

Thank you,
Nickolaus