Ungroup hangs

Hello,

My workflow includes a column with 300k+ entries in an array ([[119025985], [124201807], [71750237], [82100444], [52988261], [414532], [261391], [60199173], [68029831], [24967599], [54719802], [3005532], [125666], [104938], [6918448], [101967036], [1456], [83763], [92135637], [6604003], .....)

 

I'm trying to use ungroup to create separate rows for each value. 

 

  119025985

  124201807

  71750237

...

The node gets to 99% very quickly and then stays that way indefinitely, running on Windows 10 with 32GB; 16 GB allocated to KNIME.

How can I deal with this many values?

Best regards,

Mitch

Hello Mitch,

sorry for the delayed reply. Is it possible that the table contains several rows with 300k elements each? I have a hunch that the phenomena that the KNIME nodes seem to hang at 99% is caused by the rowid duplicate checker. The checker writes out 100k chuncks to disc which are then checked for duplicates. So if your input table contains a lot of rows with 300k elements each the result table contains several million rows which are checked for duplicates. Another factor that affects the time the duplicate checker requires is the size of the RowId. The longer the RowId the more the checker needs to write, read and compare. Since the Ungroup node uses the original RowId with the counter as suffix you might speed up the process by using the RowID node to assign a short RowId to the table you want to ungroup.

Bye

Tobias