Table connection broken when viewing sorted table

[swebb]

Hi

For a few years we’ve been hitting a bug where we get the error along the lines of:

ERROR Row Filter 0:9782 Execute failed: Blob file location “C:\development\applications\knime\knime_4.1.0_oracle-jre\col_14\000\000” does not exist

I believe there have been posts on this issue in the past but not sure where they are.

The problem appears to be that KNIME starts looking in the install directory not the workflow directory for some data.

We’ve had trouble reproducing the bug but I’ve now managed to find some steps that cause it.

The steps in my workflow that don’t trigger it:

1. Run the rule engine
2. Run the row filter

The steps in my workflow that trigger it:

1. Run the rule engine
2. Run the filter
3. View the output table
4. Sort on a column
5. Without closing the table view re-run both nodes

Having a sorted table view open triggers the bug.

KNIME log during these actions

2020-07-03 15:14:30,382 : DEBUG : KNIME-Worker-58-Rule Engine 2:9659 : : Node : Row Filter : 2:9782 : Configure succeeded. (Row Filter)
2020-07-03 15:14:30,383 : DEBUG : KNIME-Worker-58-Rule Engine 2:9659 : : NodeContainer : Rule Engine : 2:9659 : Row Filter 2:9782 has new state: CONFIGURED_QUEUED
2020-07-03 15:14:30,393 : DEBUG : KNIME-BackgroundTableWriter-1 : : Buffer : Rule Engine : 2:9659 : Buffer file (C:\Users\samuel\AppData\Local\Temp\knime_Bug36916\knime_container_20200703_8427763946115024627.bin.snappy) is 0.009MB in size
2020-07-03 15:14:30,470 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : WorkflowManager : Row Filter : 2:9782 : Row Filter 2:9782 doBeforePreExecution
2020-07-03 15:14:30,470 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : NodeContainer : Row Filter : 2:9782 : Row Filter 2:9782 has new state: PREEXECUTE
2020-07-03 15:14:30,470 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : WorkflowDataRepository : Row Filter : 2:9782 : Adding handler 599f5edd-7fda-499c-bd87-80b967282888 (Row Filter 2:9782: ) - 3 in total
2020-07-03 15:14:30,470 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : WorkflowManager : Row Filter : 2:9782 : Row Filter 2:9782 doBeforeExecution
2020-07-03 15:14:30,470 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : NodeContainer : Row Filter : 2:9782 : Row Filter 2:9782 has new state: EXECUTING
2020-07-03 15:14:30,470 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : LocalNodeExecutionJob : Row Filter : 2:9782 : Row Filter 2:9782 Start execute
2020-07-03 15:14:30,471 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : BufferSettings : Row Filter : 2:9782 : Using table format org.knime.core.data.container.DefaultTableStoreFormat
2020-07-03 15:14:30,486 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : Node : Row Filter : 2:9782 : reset
2020-07-03 15:14:30,486 : ERROR : KNIME-Worker-59-Row Filter 2:9782 : : Node : Row Filter : 2:9782 : Execute failed: Blob file location “C:\development\applications\knime\knime_4.1.0_oracle-jre\col_14\000\000” does not exist
org.knime.core.data.container.DataContainerException: Blob file location “C:\development\applications\knime\knime_4.1.0_oracle-jre\col_14\000\000” does not exist
at org.knime.core.data.container.DataContainer.checkAsyncWriteThrowable(DataContainer.java:468)
at org.knime.core.data.container.DataContainer.close(DataContainer.java:574)
at org.knime.base.node.preproc.filter.row.RowFilterNodeModel.execute(RowFilterNodeModel.java:237)
at org.knime.core.node.NodeModel.execute(NodeModel.java:737)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:571)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1236)
at org.knime.core.node.Node.execute(Node.java:1016)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:557)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:218)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:124)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:334)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:210)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: java.io.IOException: Blob file location “C:\development\applications\knime\knime_4.1.0_oracle-jre\col_14\000\000” does not exist
at org.knime.core.data.container.Buffer.getBlobFile(Buffer.java:1586)
at org.knime.core.data.container.Buffer.handleIncomingBlob(Buffer.java:1047)
at org.knime.core.data.container.Buffer.handleIncomingBlob(Buffer.java:912)
at org.knime.core.data.container.Buffer.saveBlobsAndFileStores(Buffer.java:868)
at org.knime.core.data.container.Buffer.saveBlobsAndFileStores(Buffer.java:841)
at org.knime.core.data.container.DataContainer$ContainerRunnable.run(DataContainer.java:1232)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2020-07-03 15:14:30,487 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : WorkflowManager : Row Filter : 2:9782 : Row Filter 2:9782 doBeforePostExecution
2020-07-03 15:14:30,487 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : NodeContainer : Row Filter : 2:9782 : Row Filter 2:9782 has new state: POSTEXECUTE
2020-07-03 15:14:30,487 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : WorkflowManager : Row Filter : 2:9782 : Row Filter 2:9782 doAfterExecute - failure
2020-07-03 15:14:30,487 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : Node : Row Filter : 2:9782 : reset
2020-07-03 15:14:30,487 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : Node : Row Filter : 2:9782 : clean output ports.
2020-07-03 15:14:30,487 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : WorkflowDataRepository : Row Filter : 2:9782 : Removing handler 599f5edd-7fda-499c-bd87-80b967282888 (Row Filter 2:9782: ) - 2 remaining
2020-07-03 15:14:30,487 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : NodeContainer : Row Filter : 2:9782 : Row Filter 2:9782 has new state: IDLE
2020-07-03 15:14:30,487 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : Node : Row Filter : 2:9782 : Configure succeeded. (Row Filter)
2020-07-03 15:14:30,487 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : NodeContainer : Row Filter : 2:9782 : Row Filter 2:9782 has new state: CONFIGURED
2020-07-03 15:14:30,488 : DEBUG : KNIME-Worker-59-Row Filter 2:9782 : : NodeContainer : Row Filter : 2:9782 : Bug 2 has new state: CONFIGURED

Happy to send the log file and workflow privately, just let me know where.

Disclaimer: I’ve bundled my only java distribution (oracle JRE) as we can’t use the openjdk version due to missing javafx and I have a number of internal integrations installed.

Cheers

Sam

[mlauber71]

Have you tried to put a cache node in fron of the node where the error happens?

[swebb]

The error happens at the cache node in that case

What actually seems to happen is it kills the whole branch that uses this table, trying to view tables upstream throw the error also.

[mlauber71]

@swebb

Disclaimer: I’ve bundled my only java distribution (oracle JRE) as we can’t use the openjdk version due to missing javafx and I have a number of internal integrations installed.

Would this also happen with a clean installation without these settings? Just to try and nail it down.

Also you could try and use a different path for the KNIME workspace.

Then I sometimes experienced that some KNIME nodes might somehow ‘remember’ a path that has been set before even if you reset them. Solution was to drag a fresh node from the menu and reconfigure it, though I do not see that scenario in your case.

Maybe you could provide us with a complete log file in debug mode, just to get some ideas.

[swebb]

Thanks for the further feedback

This has happened long before we had to customise the jre used, though in those days it was oracle anyway.

Even so it’s a fair suggestion to check if the error happens on a fresh knime install. I’ll have a look.

Changing the workspace location shouldn’t really matter we see this error on multiple machines.

It’s not so much miss remembering a location as it is a breakage of a link to the workflow directory which then corrupts the workflow. The workflow neeeds to be closed and reopened to be able to access the data table properly again. I’ve not spotted that issue you mention myself but will keep an eye out.

I deliberately didn’t share the whole log file and workflow publicly. If I can reproduce it on a clean knime install I’ll upload.

Cheers

Sam

P. S. Whoops seems I asked this same thing 4 years ago

Intermittent sorting error

And others seem to have a similar problem

Knime 3.4 Errors

Hey everyone, I was eager to try out the new Knime 3.4. So a few days back I installed it to test if there might be any features making chemoinformatics easier. Now I have realized that the new Software keeps failing usually with Blob errors. Either the Node content cannot be displayed or a Loop end fails with such an error.  The loop failes with: Execute failed: Error while writing to buffer: Blob file location "/home/XXX/Desktop/col_72/000/000" does not exist Whereas I get this warning i…

k-means node executable only once per session

There are some molecule structures and SDF fields. When I filter this out, it works!   The workflow in the image (contains SDF columns) just produced the following output: WARN      BlobWrapperDataCell                Blob file location "/home/user/col_19/000/000" does not exist WARN      BlobWrapperDataCell                Blob file location "/home/user/col_19/000/000" does not exist WARN      BlobWrapperDataCell                Blob file location "/home/user/col_19/000/000" does not exist …

Workflow Corruption, KNIME 3.6 & MOE

Restarting KNIME fixes the problem until you try to repeat the action. So, it worked once and then I rerun the workflow and get the error. Attach the logfileKnimeLog_afterReboot.txt (56.0 KB) Further info. It seems there was a part of the workflow that did not run correctly and generated an empty cell in an SDF column. However, the task and workflow completed successfully. After resetting that workflow and rerunning it, I do not have errors anymore.

So seems this is already logged and I’d just forgotten I’d commented on this

@marc-bux now that I have something I can reproduce simply on my machine is there anything I can do to provide some more info to help with a fix?

[swebb]

I have taken the latest version of KNIME from the downloads page. Installed the Basic chemistry types plugin and re-tested this workflow. The issue is still reproducible.

I’ve also found some new behaviour, getting the issue to trigger on the bottom branch causing the node on the top branch to ‘hang’, I don’t see anything in the log indicating the Row Sampling has crashed.

I have tried to filter the table down a bit to see the features required in the data to trigger the bug.

Still happens with 1000 rows but doesn’t happen if I remove the molfile column (which is presumably the only Blob type in the table).

I have attempted to make a publicly shareable version of the workflow by replacing the structures with some dummy ones but this new table doesn’t trigger the bug

I can send this workflow to KNIME privately though if that would help.

I’ve attached a workflow log, actions taken:

1. Load workflow
2. Reset constant val column, run both nodes
3. Reset constant val column, run both nodes
4. View Row filter table, sort on pchembl_value
5. Resent constant value column, run both nodes <- bug triggered

knime.log (25.9 KB)

Presumably links in with the temp files that are reported to be deleted?

Cheers

Sam

[mlauber71]

Have you tried to change KNIME’s internal storage to parquet or if you have that back to zip and see if this makes any difference.

KNIME Column Storage (based on Apache Parquet)

[swebb]

DEBUG NodeContainer Row Filter 0:9791 has new state: CONFIGURED
DEBUG Buffer Deleted temporary file “C:\Users\samuel\AppData\Local\Temp\knime_container_20200707_2640020351320465964.parquet”
DEBUG Buffer Deleted temporary directory “C:\Users\samuel\AppData\Local\Temp\knime_container_20200707_2640020351320465964.parquet_1”
DEBUG Row Filter 0:9791 Using table format org.knime.parquet.ParquetTableStoreFormat
DEBUG Row Filter 0:9791 Using table format org.knime.parquet.ParquetTableStoreFormat
DEBUG NodeContainerEditPart Constant Value Column 0:9789 (CONFIGURED)
DEBUG NodeContainerEditPart Row Filter 0:9791 (CONFIGURED)
DEBUG ExecuteAction Creating execution job for 1 node(s)…
DEBUG NodeContainer Constant Value Column 0:9789 has new state: CONFIGURED_MARKEDFOREXEC
DEBUG NodeContainer Constant Value Column 0:9789 has new state: CONFIGURED_QUEUED
DEBUG NodeContainer Row Filter 0:9791 has new state: CONFIGURED_MARKEDFOREXEC
DEBUG NodeContainer BugColumnStorage 0 has new state: EXECUTING
DEBUG NodeContainer ROOT has new state: EXECUTING
DEBUG Constant Value Column 0:9789 Constant Value Column 0:9789 doBeforePreExecution
DEBUG Constant Value Column 0:9789 Constant Value Column 0:9789 has new state: PREEXECUTE
DEBUG Constant Value Column 0:9789 Adding handler 3635daee-1fdf-4062-9484-5b60fdf09927 (Constant Value Column 0:9789: ) - 5 in total
DEBUG Constant Value Column 0:9789 Constant Value Column 0:9789 doBeforeExecution
DEBUG Constant Value Column 0:9789 Constant Value Column 0:9789 has new state: EXECUTING
DEBUG Constant Value Column 0:9789 Constant Value Column 0:9789 Start execute
DEBUG Constant Value Column 0:9789 Using table format org.knime.parquet.ParquetTableStoreFormat
INFO Constant Value Column 0:9789 Constant Value Column 0:9789 End execute (0 secs)
DEBUG Constant Value Column 0:9789 Constant Value Column 0:9789 doBeforePostExecution
DEBUG Constant Value Column 0:9789 Constant Value Column 0:9789 has new state: POSTEXECUTE
DEBUG Constant Value Column 0:9789 Constant Value Column 0:9789 doAfterExecute - success
DEBUG Constant Value Column 0:9789 Constant Value Column 0:9789 has new state: EXECUTED
DEBUG Row Filter 0:9791 Configure succeeded. (Row Filter)
DEBUG Constant Value Column 0:9789 Row Filter 0:9791 has new state: CONFIGURED_QUEUED
DEBUG Row Filter 0:9791 Row Filter 0:9791 doBeforePreExecution
DEBUG Row Filter 0:9791 Row Filter 0:9791 has new state: PREEXECUTE
DEBUG Row Filter 0:9791 Adding handler 653a10a6-0b74-44a2-bcc7-a65013c96fe1 (Row Filter 0:9791: ) - 6 in total
DEBUG Row Filter 0:9791 Row Filter 0:9791 doBeforeExecution
DEBUG Row Filter 0:9791 Row Filter 0:9791 has new state: EXECUTING
DEBUG Row Filter 0:9791 Row Filter 0:9791 Start execute
DEBUG Row Filter 0:9791 Using table format org.knime.parquet.ParquetTableStoreFormat
DEBUG Row Filter 0:9791 reset
ERROR Row Filter 0:9791 Execute failed: Blob file location “C:\development\applications\knime\knime_4.1.3-clean\col_0\000\000” does not exist
DEBUG Row Filter 0:9791 Row Filter 0:9791 doBeforePostExecution
DEBUG Row Filter 0:9791 Row Filter 0:9791 has new state: POSTEXECUTE
DEBUG Row Filter 0:9791 Row Filter 0:9791 doAfterExecute - failure
DEBUG Row Filter 0:9791 reset
DEBUG Row Filter 0:9791 clean output ports.
DEBUG Row Filter 0:9791 Removing handler 653a10a6-0b74-44a2-bcc7-a65013c96fe1 (Row Filter 0:9791: ) - 5 remaining
DEBUG Row Filter 0:9791 Row Filter 0:9791 has new state: IDLE

Good idea but unfortunately same error

[mlauber71]

@swebb

ERROR Row Filter 0:9791 Execute failed: Blob file location “C:\development\applications\knime\knime_4.1.3-clean\col_0\000\000” does not exist

This seems to be associated with a blob file. Could you provide us with a simple example of such a blob column/file?

[swebb]

This seems to be associated with a blob file

It is indeed and in this case it’s caused by the molfile column (as above filtering out that column the issue doesn’t happen). Walking through the code it looks like changing to parquet won’t make a difference as blob files are handled separately (and seem to be stored in a temp file).

Could you provide us with a simple example of such a blob column/file?

I’ve still yet to be able to create a table with some dummy data in this column that I can share publicly. Replacing the column with some dummy molfiles then results in the error going away. So were stuck at I can share the file privately with KNIME but not publicly at the moment.

However, I went to try debug the code to see if I could work out what’s going on. Looks like this may be fixed in the nightly build. I’m struggling to consistently recreate the issue in the nightly build of the code.