I just want to ask if you have any solution if your Knime node slows down? Currently I am using a Cross Joiner Node and I am joining five columns with 100+ rows each that made it a large data. The problem is that it slows down the execution, sometimes it does not execute at all.
Hi,
which version of KNIME are you using? KNIME 4.0 has brought many performance improvements, so you might want to update, if you haven’t already.
Maybe it is also possible to replace the cross join by a combination of some other nodes, e.g. if you do not need all the combinations of rows? If you could share the relevant workflow snippet and explain a bit, we might find a faster solution together.
Kind regards
Alexander
My Knime version is 4.0.1. Do you know any replacement for Cross Joiner? I think there’s nothing complicated about the workflow. It’s just that its slow. I mean, I use the cross joiner to make patterns. For example the first column has 10 rows with tags A1-A10, the next column has 6 rows with tags B1-B6 and the third column has 7 rows with tags C1-C7 to create patterns like A1-B1-C1, A1-B1-C2, A1-B1-C3…A10-B6-C7. The problem here is that I am cross joining columns with so many rows. If you have any other replacement for cross joiner, It would be a great help. Thanks in advance.
Hi @Gambit,
Well, if I understand correctly that should only amount to 420 rows in total (1067), which KNIME can handle pretty easily, even with a cross joiner. For general performance optimization recommendations, you can have a look at this site, especially the first tip concerning the knime.ini.
Kind regards
Alexander
The number of rows above is just an example, actually I am cross joining 5 columns that have 830, 728, 950, 590 and 856 rows. Anyway, thank you for the advice. I’ll update this topic if I found other solution.
Oh, ok That might be a bit too much. If you really need every combination, I also don’t really see a good solution for this problem. Maybe it is faster to generate it in nested loops using an R or Python node.
Kind regards
Alexander
Yes Sir That’s why it’s very slow or sometimes not executing. I really need all the combinations. I never tried using the Python Node but I will try it.
After installing the Knime Streaming, still I can’t see the "Encapsulate into Wrapped Metanode, can you give me sample workflow with Wrapped Metanode and I will try opening it on my Knime Platform?
I already made the streaming but still it’s taking so long. Is this normal for cross joining five columns with 830, 728, 950, 590 and 856 rows? I set the chunk size to 100,000. Are there any other ways or my data is just big to transfer fast?
Hi @Gambit,
by five columns, do you mean five tables? If you cross join those (so 4 cross joins), you end up with 289907101120000 (2.8 quintillion) rows. That’s definitely going to take a while.
Kind regards
Alexander