High Performance Word Combinations

KNIME Analytics Platform
#1

Hello,
I have a list of 85,000 keywords (shave, cream, skin, lotion) I need to create combinations of ALL of them (Shave Cream, Shave Skin, Shave Lotion, Cream Shave etc…)

What is the fastest way to do this? I have multiple CUDA GPUs, 512 GB of Ram etc… I am sure its some Python or something…

Thank you

#2

Hey @nxfxcom,

you could use the Cross Joiner node, so that you get all combinations in two columns. Afterwards you can use the String Manipulation node to combine the strings of the two columns into one.

forum_Combinations.knwf (10.2 KB)

Cheers,

Julian

2 Likes
#3

Hi @nxfxcom

Here is a workflow using Python packages Itertools.I returns unique combinations of 2 keywords (A,B = B,A). all_Combinations_using_python.knwf (23.2 KB)

gr. Hans

#4

Thank you! I tried them both, and even with Knime set to allow maximum memory usage 250G, all in memory and fast SSD and CPUs its taking forever and its barely using the system.

Any ideas?

#5

Cross Joiner is streamable. Try to convert it to component and stream it.
Also, load it to any DB and try cross join in DB.

1 Like
#6

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.