Knime is slowing down

KNIME Analytics Platform
1

Hi all,

lately I´m realizing that my workflows run more and more slowly. I have all my workflows in the default workspace. I run all my workflows through batch jobs and windows task manager. I have assigned 4Gb RAM memory of 8 in my system in knime configuration file. Data volume managed by the workflows hasn´t gone up significantly.
How could I boost my workflows ? Maybe creating different workspaces?

Thank you very much.

Regards

2

That is certainly odd. To the best of my knowledge, there haven’t been any recent major changes to KNIME that are known to have a degrading effect on performance. From my experience, the most dominating performance factor in KNIME is the materialization of intermediate data tables on disk.

Have you noticed the performance of your hard disk degrading independently of KNIME?

While we have some performance improvements for KNIME in the pipeline, here are some small adjustments you can try out:

  • You can disable GZIP-compression of KNIME data tables by putting -Dknime.compress.io=false in your knime.ini. Or, better yet, you can install the KNIME Column Storage (based on Apache Parquet) data store and activate it in File -> Preferences -> KNIME -> Data Storage.
  • You can increase the amount of table cells KNIME keeps in memory before writing to tables to disk by putting -Dorg.knime.container.cellsinmemory=10000000 in your knime.ini (the default value is 100000).
  • You can configure the Windows Defender to ignore the KNIME workspace and/or KNIME temp directory. Obviously, do this only (a) on your own risk, (b) if you’ve set up a dedicated KNIME temp folder, and (c) you trust all data you’re working with in KNIME.
  • Maybe try a different workspace, ideally on another hard disk.

It might also be worth having a look at a blog post from 2017 that discusses optimizing KNIME workflows for performance: https://www.knime.com/blog/optimizing-knime-workflows-for-performance

Best,

Marc

6 Likes
3

Thank you very much Marc,

I have performed some of your suggestions. I have installed Knime Column Storage and increased amount of table cells in memory. I will be carefull to watch any performance improvement.

Thank you very much again,

Regards.