[Solved] Performance Question Loop with Java Snippet combined

KNIME Analytics Platform
1

Hello Everyone,

I am trying to compare 2 different Tables against each other, so the best idea i came up with was something like the one in the Picture.  From the bottom left side Comes a Table with roughly 20.000 rows and the top left one provides me with another table with about 5.000 rows.

Now i have to check each of the 20k rows with each of the 5k ones...

my Problem is, that executing this is suuuuper slow. it takes like 30min to finish.

I thought that my code may be slow, but i only have 2 for each Loops (every table has its Information stored in an Array) plus 1 comparation..

 

The other Thing is.. the Loop end node is creating new rows instead of overwriting the old ones. is there a better way to get the results? i am just incrementing a number based on the Java snippet

 

Hope i explained my Problem somehow understandable.

 

Greetings, Jawed 

 

2

Hi Jawed, 

I would recommend you not to use loops, instead use the Cross Joiner node (takes around 10 minutes for 20k x 5k rows on my machine) and after either a Rule Engine node to compare the records or a Math Formula node to calculate difference.  

Please find attached a sample workflow. 

 

Best,

Anna

3

Hello Anna,

i was able to bypass the double table idea with another one.

Now the execution takes just around 30 sec... manipulating data while reading is so much faster 

¯\_(ツ)_/¯

I appreciate your effort and thank you for your time =)

Greetings,

jawed