Increase the speed of processing a node

[Shadi]

Hi Friends,

I have a huge data (~50 m rows) set that I am manipulating with Knime.
Running the nodes take a lot of time. Is there a way to speed this up?

[ArjenEX]

Hi @Shadi

This recently came up as well in another topic. It’s perhaps a long read but some tips and tricks are shared there as well.

Ways to speed this up? 10M + rows

Hi Folks looking for suggestions as to how i can speed this up I have a data set with normally 10M+ rows within this i have a col with anything from 5+ unique values normally not exceeding 100. For each unique value in that col i need to filter rows then perform a data integrity check [image] The row filter takes a massive amount of time due to the number of rows, and when i put this in a loop, well, that’s multiplied by no of unique values Any ideas as to how i can speed this up. I was…

[Daniel_Weikert]

@ArjenEX
are you aware of an article or docu where different file formats are compared in regards to speed by any chance?
eg parquet vs csv, vs table, …
br

[ArjenEX]

I don’t and to be honest, I’m not actively chasing those sort of things. I work with KNIME professionally and in 95% of the cases limited to the spec of the customer in terms out input and output types. If it’s really get tricky client-side for me I’ll utilize our server with 64GB VM which gets the job done as well.