Snappy: Error writing to buffer

After much soul searching with this Error writing to buffer problem, I have eventually determined it is something to do with the X-Aggregator node.
This error happens so regularly with this node.
I am on a MacBook Pro, with macOS Catalina, and now macOS Big Sur, and its the same problem, using the latest KNIME 4.3.0.

Example error:
ERROR X-Aggregator 3:61 Execute failed: Error while writing to buffer, failed to write to file "knime_container_ …

I notice as soon as I filter out any columns in SDF format the problem disappears. I can convert them to SMILES format also, and that works too.
There must be a bug somewhere in how these formats are being stored.
Simon.

One idea could be to try KNIME’s new columnar storage approach and see if now the implementation of Parquet file format was successful and night help with your problem.

Hi Simon (really pleased to see you here again!)

Do you have a workflow or some more details to share? The knime.log file would be a start but a workflow + instructions how to reproduce would even be better. Are there any partner or community extensions involved?

Have you seen this problem on systems other than Mac? We have a ticket open to work around an issue on Mac where the OS would decide to delete temporary files after ~3 days or so (maybe that also depends on the system configuration) even if the owning process is still alive.

In short: No idea, need more details.

  • Bernd
1 Like

Hi Bernd,
Good to hear from you. I’m Still using KNIME more than ever, I would be lost without it. It has totally transformed my knowledge of data science and cheminformatics over the last 10 years or so of using it! :slight_smile:
Sadly I don’t get enough time to visit the forum pages frequently.

So back to the problem. In all the time I’ve used knime I’ve never encountered these errors until using it on the Mac. I’ve tried it on 3 different MacBook Pro’s, all showing the same problem.
These are things I can tell you:
-happened since knime 4.1 onwards. It could have started before 4.1, but that was the time I switched to Mac.

  • changed the xmx memory from 2Gb right up to 12Gb, without improvement.
    -changed the default workspace to the downloads folder, and that did not improve matters.
  • MacOS Catalina and Big Sur both show the problem.
  • Tried changing from snappy to gzip and problem still present.
  • Workflow could be less than 1 hour old, and problem still happens.
  • workflows where it happens have always had an SDF data type column, and an RDKit Fingerprint column. Workflow typically also has around 20 columns of Double, Integer, and String Columns. Rows around 100 to 1000. The workflow is not especially big, maybe 20-50 nodes in total. No other partner/community nodes used, just standard knime nodes, rdkit fingerprint node, and Marvin molconverter node.
  • switching the SDF column to a SMILES column using Marvin Molconverter early in the workflow fixes the issue.
  • the fault always seems to occur on a loop end type node, especially the X-aggregator node. This error still occurs even when inside a metanode.
  • the fault does not reproduce every single time, so you can reset the whole workflow, rerun it, and most of the time it gives the error, and sometimes it does not error.
  • resetting part of the workflow and rerunning sometimes allows the workflow to run correctly. Other times you need to restart the whole workflow. You may need to restart several times until you get lucky and it works without errors.
  • if you have multiple loops in the workflow which end up running at the same time (I.e. in parallel), this results in the error pretty much every time.
  • when it errors on a loop end node, it seems to affect quite a number of nodes backstream to this. So if you try and view the data table of these prior nodes, it shows the table but as you scroll up and down, hundreds of errors are being produced in the console about problems with blobs and most of the SDF rendered molecules are missing. If you try and save the workflow, these prior nodes which had run, suddenly get red X symbols on them.

That is as much information I can produce at the moment. I will see if I can get a workflow together to reproduce it.

Simon.

4 Likes

I assume you load data from sdf via SDF Reader? Can you try to exact “Mol” instead of “SDF” and see what happens?

image

You probably know that with SDF blocks the data properties are kept inside the structure cell but not with mol blocks.

3 Likes

Is this in any way related to a similar issue on windows: Table connection broken when viewing sorted table ?

Molfile/SDF may be stored as blob cells and thus be handled via temp files. These temp files sometimes disappear resulting in a null pointer on the file and it then seems to fall back to the knime install location.

Cheers

Sam

3 Likes

Give kienerj and swebb a prize. They have solved it.
If I load in my data with the SDF Reader using “Extract SDF Blocks”, then this is when the error occurs at snappy and writing to knime_containers.
If I load in my data with the SDF Reader using “Extract Mol Blocks”, then the error DOES NOT occur. Even if I use MolConverter to convert the MOL structures back to SDF structures, the error still does not occur.
Additionally if load in my data using the “Extract SDF Blocks”, and then convert it to MOL format with the MolConverter, that also works fine.
So the bottom line, is it seems that the issue relates to the data being kept in the SDF Blocks. This is where the errors are coming from.

Well done all. Such a relief after months of this problem!

Just need a proper fix now, instead of relying on this workaround!!

Simon.

6 Likes

@wiswedel @richards99 (good to see you back on here!) @swebb - yes, I found annoying breakages with workflows when I’ve left KNIME running on W10 for a long time too, and tracked it back to temp files being deleted, so not just a Mac issue, although maybe the Mac OS is more aggressive at cleaning up?

Steve

I think these are two issues:

  1. Temporary files getting deleted by the OS. We are aware of that problem, at least on MacOS, and will fix that soon (though I would still point fingers at the OS – why on earth would it delete files associated with a program still running?). Default on Mac seems to be 3 days (search “3 days”).
  2. The problem Sam and Simon run into with SDF files represented as Blob. It would help if we could get a small-ish workflow to reproduce (maybe noting that SDF records need to be >8kB before they are stored as Blobs, otherwise they get inlined.)

– Bernd

3 Likes

I’ll see if I can get it to trigger on some public data and/or retrieve the example from my post and email it over (that one I can’t post here). Looks like I just need to try some bigger structures.

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.