Get DataTable size

Hello,

Is there a way to find the size of the table on the output?

I have around 10 million rows, and 50+ columns, and would like to estimate the memory requirement to keep that table in RAM. 

Do all data types require the same amount of memory?

Hi Joseph,

There is no straightforward way to calculate the table size beforehand since strings for example can take up a lot more space than doubles or integers.

If you want to see your current memory usage, you can go to File -> Preferences -> General and then turn on "Show heap status". In case the file size surpasses the amount of memory allocated to KNIME, the data is automatically written to disk.

If you want to increase memory allocation, open knime.ini on your local installation and add the line "-Xmx3G" at the end and replace "3" by the amount of memory in gigabytes to you want to allocate.

Best,

Roland

Hi Roland, 

Thank you for the hints.

I actually wanted to get a size of the table on the output of the node, after the node has been executed. It's pretty well behaved, so I could get an awerage per row and extrapolate based on expected table length. 

I guess I could try to forward that table to Java Snippet? 

Hi Joseph,

Glad I could help!

Yes, forwarding the table to a Java Snippet should work.

As an alternative, if you have a working R installation, you could simply forward the table to an R Snippet node and enter the following code in the configuration:

knime.out <- data.frame(obj_size = as.character(object.size(knime.in)))

This will print the size of the object in bytes to a table and pass it on to KNIME.

Best,

Roland

 

The 'Extract Table Dimension' node will also get you the table size...

Steve