KNIME data size limit

I would know what data size KNIME supports (how many registers and how many variables). Someone know it? Thanks!

KNIME's ability to handle large data sets is only limited by your available hard drive space (not the main memory!). When we realize that the data on any given connection becomes to large, we start buffering it to disk. You can actually force this to happen all the time in the context menu of a node (right click the node's icon).

To give an example: processing about 20 million images takes a few hundred giga byte of hard drive space but it works! (well, KNIME has to hold the data somewhere, doesn't it?)

- Michael

thaks Michael!

At what point does Knime decide it needs to cache the data? My Win10 PC has Knime java heap set to 32GB and still seeing slowdowns.

Where is the data cached - would it make sense to move this location to an SSD?

Cheers,

Steve.

Yes having the worksapce and temp file dir on a fast disc for sure helps a lot. If you ant to go to the extrem, put the workspace on a RAMdisk.

Another option is to put compatible nodes into wrapped metanode and use the streaming executor. This does not write out the data to disc after each node and can lead to 10-100x times speedup in worklfows with many rows.