Cross Joiner Node is slowing down

Hi Guys,

I just want to ask if you have any solution if your Knime node slows down? Currently I am using a Cross Joiner Node and I am joining five columns with 100+ rows each that made it a large data. The problem is that it slows down the execution, sometimes it does not execute at all.

Thank you in advance :slight_smile:

Hi,
which version of KNIME are you using? KNIME 4.0 has brought many performance improvements, so you might want to update, if you haven’t already.
Maybe it is also possible to replace the cross join by a combination of some other nodes, e.g. if you do not need all the combinations of rows? If you could share the relevant workflow snippet and explain a bit, we might find a faster solution together.
Kind regards
Alexander

Hi @AlexanderFillbrunn ,

My Knime version is 4.0.1. Do you know any replacement for Cross Joiner? I think there’s nothing complicated about the workflow. It’s just that its slow. I mean, I use the cross joiner to make patterns. For example the first column has 10 rows with tags A1-A10, the next column has 6 rows with tags B1-B6 and the third column has 7 rows with tags C1-C7 to create patterns like A1-B1-C1, A1-B1-C2, A1-B1-C3…A10-B6-C7. The problem here is that I am cross joining columns with so many rows. If you have any other replacement for cross joiner, It would be a great help. Thanks in advance.

Hi @Gambit,
Well, if I understand correctly that should only amount to 420 rows in total (1067), which KNIME can handle pretty easily, even with a cross joiner. For general performance optimization recommendations, you can have a look at this site, especially the first tip concerning the knime.ini.
Kind regards
Alexander

Hi @AlexanderFillbrunn,

Thank you very much for your help :slight_smile:

The number of rows above is just an example, actually I am cross joining 5 columns that have 830, 728, 950, 590 and 856 rows. Anyway, thank you for the advice. I’ll update this topic if I found other solution. :slight_smile:

Regards,
Gambit

Oh, ok :smiley: That might be a bit too much. If you really need every combination, I also don’t really see a good solution for this problem.
Maybe it is faster to generate it in nested loops using an R or Python node.
Kind regards
Alexander

Streaming :slight_smile:

The Cross Joiner is streamable, and so is the rule based row filter, those two together are a super powerfull combination.

Iff (as Alex said) you do not need all rows in the end, this will crazy speed your work.

1 Like

Hi @Iris,

Actually, I already removed the unnecessary rows :):grin: All that’s left are the ones I really need. Can I have a sample workflow?

Thank you and regards,

Gambit

Hi @AlexanderFillbrunn,

Yes Sir :wink: That’s why it’s very slow or sometimes not executing. I really need all the combinations. I never tried using the Python Node but I will try it.

Thank you very much!

Best Regards,

Gambit

HI @Gambit.

for streaming check this blog post.

Br,
Ivan

hi @ipazin,

After installing the Knime Streaming, still I can’t see the "Encapsulate into Wrapped Metanode, can you give me sample workflow with Wrapped Metanode and I will try opening it on my Knime Platform?

Thank you in advance :blush:

Hi @Gambit from KNIME 4 the Wrapped Metanode" is called “Component”.

1 Like

Hi @HansS,

Thank you for the information :slightly_smiling_face:

Regards,

Gambit

1 Like

HI there,

@Gambit if you will still have issues I can create sample workflow for you.

tnx @HansS!

Br,
Ivan

1 Like

Hi @ipazin,

I already made the streaming but still it’s taking so long. Is this normal for cross joining five columns with 830, 728, 950, 590 and 856 rows? I set the chunk size to 100,000. Are there any other ways or my data is just big to transfer fast?

Thank you in advance

Hi @Gambit,
by five columns, do you mean five tables? If you cross join those (so 4 cross joins), you end up with 289907101120000 (2.8 quintillion) rows. That’s definitely going to take a while.
Kind regards
Alexander

1 Like

Hi @AlexanderFillbrunn,

Yes :sweat_smile:

Regards,

Gambit

1 Like

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.