Need for a db sql connector node to wait for a completion of a long data manipulation


I have a workflow that takes a long time to do some data manipulation (2+ hours), due to data volumes involved and complexity of rules. At the end of this long data manipulation I have to do 2 things:

  • truncate a db table
  • rewrite the same db table with the new data.
    This is done by 3 nodes in sequence:
    1*) PostgreSQL Connector
    2*) DB SQL Executor, which truncates the target db table
  1. DB Insert, which inserts the new data in the db table.

Now, the Connector and DB SQL Executor nodes have no predecessors, so they start as soon as I execute the workflow. As a consequence I have 2 problems:

  • the target table remains empty for a long time
  • when the DB Insert node receives the input data, the db connection is lost and the node execution hangs.

I suppose I have to add some predecessor in order to put the nodes 1*) and 2*) (see above) in the queue of the data manipulation. I tried with flow variables, but for some reason I’m not able to connect to the variable ports of the nodes 1*) and 2*).

I also tried to use the wait node but the “wait for time” / “wait to time” / “wait for file” methods are not suitable in my case.

Could anyone suggest a way to resolve this problem?
Thanks a lot

I think you should try again to use Flow Variables. What you also could do is try to configure the DB connector node to reconnect or have a longer timeout (maybe sse the description).

1 Like

I tried again, now it works.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.