Does anyone know if there is a way to make the workflow saving process any faster? I have a relatively simple workflow that’s loading an excel file with ~4500 rows, 4.7mb in size, then I have a handful of nodes to basically create a manual Monte Carlo by randomly drawing 1,000 rows, 500 times. The workflow executes in about 20 seconds, but takes about 20-25 minutes to save. Has anyone else encountered this and is there a setting that I can adjust to make this any better?
Hi there @ekostic,
welcome to KNIME Community Forum!
your saving time takes really long. How does your environment looks like - OS? Some security mechanism / virus scanner? Workspace location?
I’m running this on Windows 10 with an Intel i7 processor, 16 GB RAM, and I do have Symantec anti-virus, though I don’t believe it’s running against KNIME each time I save. I did just check my workflow file size and I am near 200 MB after the 500+ iterations of random row sampling (1,000 times each)… so my 4,500 row file has turned into over 500 1,000-row files… so my guess is that’s slowing it way down. I still think 20+ minutes to save seems long as I would imagine others using this have files with over a million rows.
I don’t need it to actually hold the ‘big’ data in the middle, I just want the end result which is a concatenation of aggregation of the sampling (which is only a few hundred rows and about 10 columns). So if there’s a way to run the workflow then essentially delete the intermediate data that could likely help a lot. Is there a way to do that?
Thank you for your help!
Don’t think there is a way to do that. Also as you said KNIME workflows handled much more data and didn’t have this issue. How long does saving just a basic workflow with couple of nodes last?
I would definitely check that anti-virus. See here: Saving a workflow with 1 node takes more than 1 minute (thread dumps attached)
After reviewing your referenced link and re-checking, it looks like my anti-virus software was significantly causing the hold-up. For anyone else encountering this with little information technology background, I was checking system performance as my workflow was saving and Symantec was not showing up as doing anything processor-heavy. As soon as I shifted my workspace folder to scheduled and on-demand it exponentially sped-up the processing within KNIME.
Glad it is working normally now
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.