Implementing a long list of queries in KNIME. Each query is used to create a table. Used this basic logic:
Database Table Connector ---> Database Connection Table Writer
To write each table back to the schema because these intermediate tables are used to create more tables down the road.
I use the com.cloudera.impala.jdbc4.Driver and connect to a secure Impala database.
All works, but even with the knime.ini parameters added:
-Dorg.knime.container.cellsinmemory=1000000
-Xmx16g
-Dknime.database.fetchsize=1000000
some Database Connection Table Writer nodes will take more than 30 minutes to finish executing, while others that handle a similar amount of data finish executing in less than a minute.
What can I do to speed this up? The queries themselves execute rapidly in Hue.
Thanks for the help.