Run Knime workflow fast machine

KNIME Analytics Platform
1

Hello
We have a Knime workflow that require more than 1 month to get the results when running in a regular computer. How one can run the same workflow in a faster machine ?

Malik

2

Hi @malik,
1 month? Really?

Well… unless you are running this workflow on a very very old (+10y) machine i don’t think that you’ll benefit so much by changing the hardware…

Can you give some more details of your workflow, like data source types, n° of tables and rows… it would help to analyze your problem.

Luca

3 Likes
3

Hi @Luca_Italy
Our table consist from 12524474 records. Because that it took so much time to process.

Malik

4

What kind of workflow is it? What nodes are you using?

I’d say look at workflow optimization because it’s indeed insanely long for 12 mil records which should bother KNIME that much.

The more you can share the better :wink:

5

What about running the workflow in AWS server ?

6

If you rent a supercomputer sure, but I’m afraid that nobody can give you an educated answer without you providing more details.

5 Likes
7

Hi @malik, without knowing anything about your particular setup, the performance is likely going to depend on a mix of factors:

  • Type and generation of CPU
  • Available memory
  • Type/Speed of hard drive (more important when memory is lower)
  • The type of processing (computation intensive, or i-o intensive, or high data volumes)
  • The way the workflow is written (types of nodes used, and performance bottlenecks).

12.5 million records in “over a month” equates to about 4 records per second, sustained, which isn’t exactly rapid.

What CPU does your machine have, and what generation of processor? (As was noted recently in a separate thread, the processor can have a dramatic effect if you are performing lots of computation)

How much memory do you have and how much is KNIME configured to be allowed to use?

Does the workflow contain many loops? Are they nested? Can any loops be replaced with alternative solutions?

Have you found key areas of the workflow that are really slow. Ideally you’d optimise as much as you can first and then look at hardware. A poorly performing workflow on faster hardware is still poorly performing but just less noticeable :wink:

Finally, back to your specific question… when you ask how you can run the workflow in a faster machine, are you asking how to physically export and import a workflow, or something else? Because the obvious answer to me is “export the workflow and import it on a faster machine”, but I’d doubt that this is what you were really asking.

As others above have said, the more information you can provide, the easier it will be to make suggestions.

1 Like
8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.