Knime Performance and Extensive Folder Structure

Maybe this has been addressed but I could not find it via a search of forum -

Knime uses a tremendous number of folders to store a work flow (it can easily be 500-1000+).  This results in very serious performance problems when you try to save or copy workflows, add new workflow groups (when it refreshes workspace), etc.  These operations can take many minutes.  The servers also are not really happy with having so many files and folders in their directory structure and I suspect this is one of the main causes of the performance problems. 

To me this is one of the biggest issues with Knime.  Has anyone else observed this and are there any solutions?  Is this an inherent characteristic of the underlying Eclipse architecture?

Mark

Dear Mark, Did you find out more about minimizing the number of folders to store workflows? I'm really concerned about performance issues because we are testing KNIME to eventually adopt it. ~thanks, Dani

No I did not find out anything more and no one from the community replied to my post.  I would be very interested to hear from the Knime developers what they know about the large number of folders and any issues and solutions.

The performance problem is not with Knime itself - in general I am amazed at how good it is though I dont usually do "big data".  The issue is with copying Knime workflows etc but I did find that there is a way to export and import and that goes better as I think it Zips them.

 

Indeed everything is zipped.

Still your biggest worry is going to be the data saved together with a workflow.
If you save a workflow _with_ the data it will save all output of every single node in the workflow. That can be a LOT of data.
You can save a workflow without its data, and that will make the eventual saved stuff quite reasonable in size. Ofcourse you will need to rerun the entire workflow to get back your data, but that should be obvious.

 

By the way, in the newer versions of Knime you can do operations like copying a workflow from the stored-workflows tree.
Besides, there is also the option to export a workflow. There it will allways ask you wether you want to include the data. The result is 1 (one) zipfile.