Memory policy with 256 Go RAM

KNIME Analytics Platform
#1

Hi everybody,

I use a parallel chunk loop to make 20 differents actions (per line) on 100 M of lines

The server is use have 256 Go RAM, il also force le maximum open files to 65536.

I dont know what is the optimum value of "chunk count" and if i must force memory policy to "keep all in memory"

Do you have any idea ?

FJ

 

#2

I don't know what will be optimal but I think the defaults are OK, i.e. it splits based on the number of available cores in your system (slightly overestimating...).

You shouldn't use too many parallel chunks as combining the results is expensive. 

I guess some experiments with binary search for optimal values may help. Interested in hearing what you learn!

 

#3

if i use chunk count around 50/70 the nodes between start and stop are faster

but it seems that start chunk node is slower, stop chunk node is very long (last aggregation)

if chunk count is around 4, start and stop are less slower but all nodes between start and stop are slower

I dont see any improvment by using memory policy - keep all in memory

I stil investigate

Regards

FJ