Optimizing Data Retrieval and Processing in Knime

Hello everyone, seeking advice on optimizing data retrieval and processing in Knime. Should I filter data within Knime after fetching from the database, or filter directly in the query? Additionally, for operations like GROUP BY and averaging, is it better to perform them in Knime or in the database itself? Your insights and experiences are much appreciated!

@albertofleitas welcome to the knime forum. This would depend on the database and the size of the data. In general it will be better to do heavy ETL work on a database. The knime sql nodes can help with that.

Also depending on the type of database you might want to plan your filters and depending on your task create intermediate tables instead of putting everything in one combined SQL view or query (which is what the brown sql connectors will do).

2 Likes