Knime Temp Folder Data not freed up?

[mwiegand]

Thanks for your quick response. What I meant is that Knime, by default, creates a sub-folder in the systems temp-dir to prevent that messiness I noticed which makes it difficult for me to identify the deprecated Knime temp data.

My primary motivation, alongside keeping thigs tidy, is to improve Knimes reliability but also find a possible explanation for the performance regression that others and I reported many times over like that recent post here where I believe to understand what is happening:

Any ideas when Parrallel Chunk loop end performance will be resolved

Thinking about the issue I start to get a good picture of what might happens. The issue is described as follows: Slow save or data collection operation i.e. for Parallel Chunk End No significant CPU, nor RAM nor Disk utilization Moving the Workspace to a separate disk shows no improvement Disc “C” utilization during save / data collection operation lingering Issue emerged (perceived wise) with late 4.x but latest version 5.x of Knime I currently have a crawl running and noticed, while also re…

PS: Sorry for the side question but what is your take on the idea to prevent Knime from writing any temp data to disk i.e. by utilizing a ramdisk? I also tried, while investigating about the perf-regression, to disable compressions and various other args in the knime.ini but none showed any significant improvement except using the ramdisk.

In Memory Computation using Ramdisk

Hi, A colleague asked my why, when attempting to bring my system to the limit to find the most efficient solutions, I am not using a Ramdisk. It’s been years I used one and on Unix but not Windows so his idea struck me like lightning (Kudos to him!). Using ImDisk I created one of 16 GB size. Exported a test workflow, you find both linked below, and created a new test workspace. This approach is like a MySQL Server and indeed blew me away. Crunching the 10 Million rows is by an order of ma…

PPS: Is there any way to prevent, whenever globally for Knime or individually per workflow, saving any temp data? The workflow that currently runs utilized Don’t Save Start/End nodes

I also write the results after each iteration into data in the workflow data directory. Still it writes so much data to disk that I am worried about what happens when the parallel chunk end node collects the results.

PPPS: Knime seems to have crashed but still has some CPU usage. I aborted the process but the temp data is not getting removed

A few minutes later no change

Cleaning the Temp data does not work using the Windows cleanup. The Temp folder in the summary vanished after click on remove but upon refresh it appears again with no change to the disk space.

Windows Cleanup, even if it ran with admin privileges that my user has anyway, no file was deleted. I cleaned the entire temp folder manually which took about 10 minutes and shows some significant CPU utilization:

Though, there was one file that was not able to get deleted.

Whilst I can uncheck the read only and apply it, it get reverted immediately despite no Knime process running in the background.

@qqilihq upon inspecting the binary files I noticed they all seem to be generated by Selenium.

Upon ending the web driver, the folder are still present in the temp directory and running the garbage collector, just in case it triggers something, as well as saving and closing the workflow did not have any effect. Only upon closing Knime the irrelevant temp data got deleted.

doing further debugging I noticed that, even though just one browser was opened, two in size identical folder got created. That lead me to suspect the Network Dump Start and End Nodes. After removing them, no temp data got created anymore.

Apologize for the messy comment but my brain is on fire trying to get to the root of this. Two more things, I added these options --disk-cache-size=0 and --incognito.