Any ideas when Parrallel Chunk loop end performance will be resolved

[Gavin_Attard]

Hi @knime

Is there a view as to when the parallel chunk loop end performance issue will be resolved. As soon as i hit large (>1M rows) the chunk end takes absolutely ages to finish and i not disk usage is sooo slow from knime.

much love

Gavin

[mwiegand]

Hi @Gavin_Attard,

thanks for raising this again. I recalled and found our prev. conversations and it seems that, given how much time has passed, it is unlikely to happen. I suppose this is too specific, though.

Ways to speed this up? 10M + rows

Hi Folks looking for suggestions as to how i can speed this up I have a data set with normally 10M+ rows within this i have a col with anything from 5+ unique values normally not exceeding 100. For each unique value in that col i need to filter rows then perform a data integrity check [image] The row filter takes a massive amount of time due to the number of rows, and when i put this in a loop, well, that’s multiplied by no of unique values Any ideas as to how i can speed this up. I was…

However, I happen to have done some more in depth investigations in the past and found that Knime performs painfully slow when it comes to certain processes related to reading and writing data. The Parallel Nodes, as well as saving a workflow, falls right in that category.

Saving Workflow takes extreemly long in 5.1

Hi, with the recent update to 5.1 I happen to notice that saving a workflow requires considerably more time. I had already saved and restarted a few times but the overall performance during a save seems to be considerably slower. The workflow in question took about a minute to start saving and than progressed rather slowly and with inconsistent saving times upon each save compared to the previous Knime version. [image] The workflow does only contains very basic text based data, though. abou…

In Memory Computation using Ramdisk

Hi, A colleague asked my why, when attempting to bring my system to the limit to find the most efficient solutions, I am not using a Ramdisk. It’s been years I used one and on Unix but not Windows so his idea struck me like lightning (Kudos to him!). Using ImDisk I created one of 16 GB size. Exported a test workflow, you find both linked below, and created a new test workspace. This approach is like a MySQL Server and indeed blew me away. Crunching the 10 Million rows is by an order of ma…

Unable to abort processing, Workspace blocked & requiring reboot

Hi, In order to back test improvements I created a workflow to exponentially increase the data (1 Row with some JSON). I encountered the issue before that Knime cannot finish due to the amount but I am equally unable to abort the processing. Whilst I am able to close Knime and no background process is running, it seems some sort of lock file is still in place blocking the workspace. Hence, I am forced to restart my whole system. [image] Best Mike

CPU and Ram are almost never going to become a bottle neck but the disk certainly is. Even if I use a high performance SSD, directly hooked up to the CPU PCIe Lanes as a separate disk, Knime can become painfully slow.

I.e. I noticed, when extracting the system properties, that the Java Temp dir is on the primary disk even if the workspace is not:

Anyhow, when saving or data is being collected like with the Parallel Chunk End node, SSD throughput is chilling at 6 % or less. To my dismay, no one felt this worthwhile having a look.

Best
Mike

[Gavin_Attard]

Agreed i also think the primary issue here is disk i/o. Its not hardware related, defo software.

[mwiegand]

Fully agree. We could certainly rule out hardware limitations and I believe we concluded, backed up by the experiences of others too, that this regression started some version of Knime ago.

Regardless of the hardware, I also managed to rule out the backend type (row vs. columnar), compression of data during save (absolutely no CPU impact) and other things too.

The only continuity I see is large amount of data which I believe Knime is trying to sort. While reading the highly recommendable article from @mlauber71

KNIME Batch Processing on Windows and MacOS

Some args caught my attention like -XX:+UseStringDeduplication or -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/path/to/gc.log. I wonder if there are processes like the string deduplication or garbage collection at fault causing the performance regression?

On second thought that would make sense as both I assume are not parallelizable using multiple CPU and the experienced issue scaled, gut feeling wise, linear with the amount of data. On top of that, it can be kind of replicated by triggering a garbage collection.

Eventually the flags in the article could help to increase log verbosity but without some support by the @knime team we would just do guesswork.

[Luca_Italy]

Holy crap.
Have been months that i’m yelling to optimize performance over new features.
Loops in general, QOL improvements (like search for field in every nodes).
Knime is a wonderful swiss knife, but is struggling to be a very good ETL platform for such reasons.

[Gavin_Attard]

Agreed, i had a client in the market, but i could not recommend it (they needed server), due to the write to disk times that are slowing everything down.

@knime this really needs addressing