Combined KNIME container read error / String Manipulator glitch in KNIME 4.4.0 (Windows) after multiple system suspends

[Ergonomist]

Dear KNIMErs,

Coming back here after a long hiatus to report my latest, likely difficult-to-reproduce glitch discovered, and to come up with (maybe/hopefully) a workaround solution.

Let me start with what I believe could be a solution: If KNIME used a dedicated temp / swap space outside of Windows’ usual Temp folders, the situation below seems unlikely to repeat. Just a feeling I get from these other threads. :))

Before I explain how the issues arose, some context up-front: In spite of being 99% of my time on Windows, I reboot my machine way too infrequently, preferring to suspend it to disk instead and picking up where I left off. It’s also easy for me to run out of system memory due to heavy, varied workloads, so presumably my swap file and temp space gets rewritten and/or overwritten way too often as well.

Though what I describe below has happened on my laptop, which does not really have serious HDD space limitations right now. But at the same time it’s the system which is most affected and at risk, since it mostly connects via slow Wifi (i.e. less complete OneDrive backups), and really only gets used as a “filler machine” when required between main desktop sessions.

I only describe this in so much detail because I believe it’s related to the (combined) issue – one root issue and one issue related to the recovery actions undertaken to repair the effects of the original issue.

So what happened? Essentially the following:

1. I work intensively on a KNIME workflow on my laptop during a business trip
2. I suspend the system with KNIME and the workflow open
3. I go on holidays and use my laptop for other things, but KNIME stays open
4. I suspend my machine again, and repeat steps 3 and 4 a few times
5. I finally return to work (today :/) and take my laptop, wanting to resume work of the KNIME workflow
6. I get the glitch below:

Execute failed: Cannot read file "knime_container_20200301_1905559478659870280.bin.snappy"

Hi @qqilihq, This second error you’ve been posting looks like an unrelated issue that has recently been brought to our attention via our automated tests. It could occur in the rare case when a table was cleared / discarded exactly at the moment when it was just finished being written to disk by some asynchronous process. The reason why you are seeing this in a loop is that here tables are frequently cleared shortly after they have been created / written to disk. It has since been fixed on our n…

Effectively I can inspect the outputs of executed “green” nodes, but I cannot add additional nodes to any existing branch because of the glitch. I cannot save this state either, because the (temp) containers on the file system are gone / inaccessible. Unsure of what’s actually the last saved state, I decide to try a recovery.

7. Since data was loaded from local Excel files, I simply copy/paste the entire workflow into itself with an intention to re-execute. This works flawlessly for a wide variety of ETL nodes including filters, renames, column merges, GroupBys, charting, etc. In one branch, however, a String Manipulator node falls over irrecoverably like here:

String Manipulator error: Unable to compile expression

Hi, I have seen this behavior now for quite some time (also before 2.10), also on different computers, and I wonder what goes wrong here.... - String Manipulator sets to "red" and states: WARN      String Manipulation                Unable to compile expression  ...and lists some java lines that cannot be resolved - now, saving the workflow in that status and restarting KNIME, will lead to "yellow" state, and node can be executed normally. - sometimes this happens to be while being in the …

So both the original issue in 6) and the String Manipulator issue in 7) in the given usage context seem to suggest a temp space disappearance owed to Windows swapping, clearing swap and temp space, etc.

I can probably now save my mostly-executed WF copy (have yet to try :)), restart KNIME and execute that String Manipulator node to recover the original workflow state. However, had KNIME used a dedicated, “untouchable” swap/ temp space, nothing should have happened in the first place… or so I think.

Could this be achieved somehow?

Many thanks,
E

[ScottF]

Hi @Ergonomist -

Thanks for the detailed post. I wanted to highlight this bit:

@Ergonomist

Let me start with what I believe could be a solution: If KNIME used a dedicated temp / swap space outside of Windows’ usual Temp folders, the situation below seems unlikely to repeat.

You can change the temp file directory that KNIME uses in File → Preferences → KNIME. Given your usage pattern, it would be interesting if you could change that, use KNIME as usual for a while, and report back about whether the problems you describe still occur.

[Ergonomist]

Hi @ScottF,

Very useful indeed, I’ll try that – thanks!

Cheers,
E

P.S.: My “usecase”, i.e. mad laziness… thanks for your diplomacy. :))