Workflow Corruption, KNIME 3.6 & MOE

joshuahoran · August 25, 2018, 2:46am

HI all,

I recently made two changes to my KNIME environment. I updated to the most recent version of KNIME 3.6 and I downloaded and installed the CCG MOE nodes. I’m observing that after you use a few of the CCG nodes, KNIME becomes unstable and fails when you try to save workflows. Upon restarting KnIme and re-opening the workflow, many nodes (which were saved in previous successful workflow saves) will simply have vanished. This is obviously devastating as it destroys existing workflows and can erase hours of work that you thought was saved.

I’m guessing the CCG nodes are not compatible with the new version of KNIME, but I want to put this out there and see if anyone else is seeing the same thing.

The errors I get when saving have this flavor:

2018-08-24 22:28:20,418 : DEBUG : ModalContext : FileSingleNodeContainerPersistor : : : Replaced node directory “C:\Users\Josh\knime-workspace*NODE_NAME*\GroupBy (#187)”
2018-08-24 22:28:20,418 : DEBUG : ModalContext : Buffer : : : Using table format org.knime.core.data.container.DefaultTableStoreFormat
2018-08-24 22:28:20,465 : DEBUG : ModalContext : Buffer : : : Buffer file (C:\Users\Josh\AppData\Local\Temp\knime_container_20180824_4380846535226257778.bin.gz) is 0MB in size

…repeat a lot of the above. Then:

2018-08-24 22:28:20,481 : DEBUG : ModalContext : InplaceSaveRunnable : : : Could not save workflow
java.lang.RuntimeException: Error while writing to buffer, failed to write to file “knime_container_20180824_8035606566907967169.bin.gz”: Blob file location “C:\Program Files\KNIME\col_1\000\000” does not exist
at org.knime.core.data.container.Buffer.addRow(Buffer.java:629)
at org.knime.core.data.container.Buffer.addToZipFile(Buffer.java:1595)
at org.knime.core.data.container.ContainerTable.saveToFile(ContainerTable.java:194)
at org.knime.core.data.container.RearrangeColumnsTable.saveToFile(RearrangeColumnsTable.java:620)
at org.knime.core.node.BufferedDataTable.save(BufferedDataTable.java:545)
at org.knime.core.node.BufferedDataTable.save(BufferedDataTable.java:540)
at org.knime.core.node.BufferedDataTable.save(BufferedDataTable.java:540)
at org.knime.core.node.FileNodePersistor.saveBufferedDataTable(FileNodePersistor.java:1448)
at org.knime.core.node.FileNodePersistor.savePort(FileNodePersistor.java:1354)
at org.knime.core.node.FileNodePersistor.savePorts(FileNodePersistor.java:1258)
at org.knime.core.node.FileNodePersistor.save(FileNodePersistor.java:1212)
at org.knime.core.node.workflow.FileNativeNodeContainerPersistor.save(FileNativeNodeContainerPersistor.java:520)
at org.knime.core.node.workflow.FileSingleNodeContainerPersistor.save(FileSingleNodeContainerPersistor.java:605)
at org.knime.core.node.workflow.FileWorkflowPersistor.saveNodeContainer(FileWorkflowPersistor.java:2260)
at org.knime.core.node.workflow.FileWorkflowPersistor.saveContent(FileWorkflowPersistor.java:2022)
at org.knime.core.node.workflow.FileWorkflowPersistor.save(FileWorkflowPersistor.java:1938)
at org.knime.core.node.workflow.WorkflowManager.save(WorkflowManager.java:8244)
at org.knime.workbench.editor2.InplaceSaveRunnable.save(InplaceSaveRunnable.java:92)
at org.knime.workbench.editor2.AbstractSaveRunnable.run(AbstractSaveRunnable.java:104)
at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:119)
Caused by: java.io.IOException: Blob file location “C:\Program Files\KNIME\col_1\000\000” does not exist
at org.knime.core.data.container.Buffer.getBlobFile(Buffer.java:1393)
at org.knime.core.data.container.Buffer.handleIncomingBlob(Buffer.java:856)
at org.knime.core.data.container.Buffer.handleIncomingBlob(Buffer.java:746)
at org.knime.core.data.container.Buffer.saveBlobsAndFileStores(Buffer.java:694)
at org.knime.core.data.container.Buffer.addRow(Buffer.java:602)
… 19 more
2018-08-24 22:28:20,497 : ERROR : main : WorkflowEditor : : : Could not save workflow: Could not save workflow: Error while writing to buffer, failed to write to file “knime_container_20180824_8035606566907967169.bin.gz”: Blob file location “C:\Program Files\KNIME\col_1\000\000” does not exist
2018-08-24 22:28:20,497 : DEBUG : main : WorkflowEditor : : : Could not save workflow: Could not save workflow: Error while writing to buffer, failed to write to file “knime_container_20180824_8035606566907967169.bin.gz”: Blob file location “C:\Program Files\KNIME\col_1\000\000” does not exist
java.lang.reflect.InvocationTargetException
at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:398)
at org.eclipse.jface.dialogs.ProgressMonitorDialog.run(ProgressMonitorDialog.java:483)
at org.eclipse.ui.internal.progress.ProgressMonitorJobsDialog.run(ProgressMonitorJobsDialog.java:237)
at org.eclipse.ui.internal.progress.ProgressManager.run(ProgressManager.java:989)
at org.knime.workbench.editor2.WorkflowEditor.saveTo(WorkflowEditor.java:1633)
at org.knime.workbench.editor2.WorkflowEditor.doSave(WorkflowEditor.java:1743)
at org.eclipse.ui.internal.SaveableHelper.lambda$0(SaveableHelper.java:154)
at org.eclipse.ui.internal.SaveableHelper.lambda$3(SaveableHelper.java:271)
at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:437)
at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:353)
at org.eclipse.ui.internal.WorkbenchWindow$14.run(WorkbenchWindow.java:2195)
at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:70)
at org.eclipse.ui.internal.WorkbenchWindow.run(WorkbenchWindow.java:2191)
at org.eclipse.ui.internal.SaveableHelper.runProgressMonitorOperation(SaveableHelper.java:278)
at org.eclipse.ui.internal.SaveableHelper.runProgressMonitorOperation(SaveableHelper.java:259)
at org.eclipse.ui.internal.SaveableHelper.savePart(SaveableHelper.java:158)
at org.eclipse.ui.internal.WorkbenchPage.saveSaveable(WorkbenchPage.java:3917)
at org.eclipse.ui.internal.WorkbenchPage.saveEditor(WorkbenchPage.java:3931)
at org.eclipse.ui.internal.handlers.SaveHandler.execute(SaveHandler.java:54)
at org.eclipse.ui.internal.handlers.HandlerProxy.execute(HandlerProxy.java:291)
at org.eclipse.ui.internal.handlers.E4HandlerProxy.execute(E4HandlerProxy.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.e4.core.internal.di.MethodRequestor.execute(MethodRequestor.java:55)
at org.eclipse.e4.core.internal.di.InjectorImpl.invokeUsingClass(InjectorImpl.java:305)
at org.eclipse.e4.core.internal.di.InjectorImpl.invoke(InjectorImpl.java:239)
at org.eclipse.e4.core.contexts.ContextInjectionFactory.invoke(ContextInjectionFactory.java:132)
at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.execute(HandlerServiceHandler.java:152)
at org.eclipse.core.commands.Command.executeWithChecks(Command.java:494)
at org.eclipse.core.commands.ParameterizedCommand.executeWithChecks(ParameterizedCommand.java:487)
at org.eclipse.e4.core.commands.internal.HandlerServiceImpl.executeHandler(HandlerServiceImpl.java:210)
at org.eclipse.ui.internal.handlers.LegacyHandlerService.executeCommand(LegacyHandlerService.java:390)
at org.eclipse.ui.internal.actions.CommandAction.runWithEvent(CommandAction.java:151)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:565)
at org.eclipse.jface.action.ActionContributionItem.lambda$5(ActionContributionItem.java:436)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:86)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4428)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1079)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4238)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3817)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$5.run(PartRenderingEngine.java:1150)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1039)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:153)
at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:680)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:594)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:148)
at org.knime.product.rcp.KNIMEApplication.start(KNIMEApplication.java:144)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:653)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:590)
at org.eclipse.equinox.launcher.Main.run(Main.java:1499)
Caused by: org.eclipse.swt.SWTException: Invalid thread access
at org.eclipse.swt.SWT.error(SWT.java:4533)
at org.eclipse.swt.SWT.error(SWT.java:4448)
at org.eclipse.swt.SWT.error(SWT.java:4419)
at org.eclipse.swt.widgets.Widget.error(Widget.java:482)
at org.eclipse.swt.widgets.Widget.checkWidget(Widget.java:373)
at org.eclipse.swt.widgets.ToolItem.setEnabled(ToolItem.java:642)
at org.eclipse.jface.action.StatusLine.setCanceled(StatusLine.java:482)
at org.eclipse.jface.action.StatusLineManager$1.setCanceled(StatusLineManager.java:177)
at org.eclipse.core.runtime.ProgressMonitorWrapper.setCanceled(ProgressMonitorWrapper.java:139)
at org.eclipse.ui.internal.dialogs.EventLoopProgressMonitor.setCanceled(EventLoopProgressMonitor.java:152)
at org.knime.workbench.editor2.AbstractSaveRunnable.run(AbstractSaveRunnable.java:133)
at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:119)

marc-bux · October 19, 2018, 8:53am

Hi Josh,

It looks like KNIME attempts (yet is unable to) create a folder for a Blob file in “C:\Program Files\KNIME”. While I’m not 100% sure, I suspect KNIME doesn’t have write permissions in said folder. You could confirm this by running KNIME as administrator, but you’d clutter your “C:\Program Files\KNIME” directory, which you probably don’t want.

In any case, could you check the directory for temporary files in “File -> Preferences -> KNIME” as well as your workspace location (the top entry in “File -> Switch Workspace”)? If neither of the two point towards “C:\Program Files\KNIME”, we’ll have to find out why KNIME tries to place create a folder in there.

Kind regards,

Marc

joshuahoran · November 16, 2018, 12:02am

Hi Marc.

I checked the locations you mentioned. The temp directory in preferences points toward: …\AppData\Local\Temp\

And “File -> Switch Workspace menu has no entries except “Other…”

marc-bux · November 19, 2018, 2:00pm

Hi Josh,

Thanks for your response. It looks like you’re temp directory and workspace location are in fact not the issue here.

After having looked some more into possible causes for the error with my colleagues, we think that it is most likely due to the directory for temporarily storing blob files being reset at some point during execution. This could cause KNIME to attempt to save blobs in KNIME’s installation folder at C:\Program Files\KNIME, for which it doesn’t have any write privileges. Unfortunately, I haven’t been able to reproduce the issue yet and I am unsure about how this is connected to the MOE nodes.

Does the issue reliably occur if you follow a deterministic sequence of steps? If so, could you provide me with an (ideally minimal) example of a workflow where this error occurs?

Thanks in advance!

Best,

Marc

swebb · November 27, 2018, 2:29pm

I don’t think this bug is related to the MOE nodes. We don’t have them installed and every now and then we get this issue come up and have never been able to identify what causes it.

Happens for different users with different workflows.

Error may be slightly different as it doesn’t destroy our workflows a node will simply throw an exception and cancel execution.

An example log output:

ERROR Recursive Loop End 0:1547:1528 Execute failed: Error while writing to buffer, failed to write to file “knime_container_20181127_2015124689891645395.bin.gz”: Blob file location “C:\Users\JamesM\knime_3.5.3\col_4\000\000” does not exist
DEBUG Recursive Loop End 0:1547:1528 Execute failed: Error while writing to buffer, failed to write to file “knime_container_20181127_2015124689891645395.bin.gz”: Blob file location “C:\Users\JamesM\knime_3.5.3\col_4\000\000” does not exist
java.lang.RuntimeException: Error while writing to buffer, failed to write to file “knime_container_20181127_2015124689891645395.bin.gz”: Blob file location “C:\Users\JamesM\knime_3.5.3\col_4\000\000” does not exist
at org.knime.core.data.container.Buffer.addRow(Buffer.java:629)
at org.knime.core.data.container.DataContainer.addRowToTableWrite(DataContainer.java:499)
at org.knime.core.data.container.DataContainer.addRowToTable(DataContainer.java:778)
at org.knime.base.node.meta.looper.recursive.RecursiveLoopEndNodeModel.execute(RecursiveLoopEndNodeModel.java:150)
at org.knime.core.node.NodeModel.execute(NodeModel.java:733)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1177)
at org.knime.core.node.Node.execute(Node.java:964)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:561)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: java.io.IOException: Blob file location “C:\Users\JamesM\knime_3.5.3\col_4\000\000” does not exist
at org.knime.core.data.container.Buffer.getBlobFile(Buffer.java:1393)
at org.knime.core.data.container.Buffer.handleIncomingBlob(Buffer.java:856)
at org.knime.core.data.container.Buffer.handleIncomingBlob(Buffer.java:746)
at org.knime.core.data.container.Buffer.saveBlobsAndFileStores(Buffer.java:694)
at org.knime.core.data.container.Buffer.addRow(Buffer.java:602)
… 17 more

joshuahoran · December 18, 2018, 10:00pm

I confirm that I cannot identify any deterministic way to reproduce this behavior. For now I have learned to watch for this problem. If I observe any early behavior that suggests this problem is manifesting, I perform a “Save As…” (not save since things might already be corrupted at that point) and then restart KNIME.

marc-bux · December 19, 2018, 4:20pm

We’ve created a ticket for this bug and will look for a solution. In the meantime, do let us know if you make any further observations that could help us in narrowing down the scenarios in which this bug appears.

Thanks

Marc

dasturiol · October 22, 2019, 2:14pm

Hi,

I recently came across this bug. I am using KNIME 4.0.0 and this happens with the “Duplicate filter row” node but only when I select the option " Remove duplicate rows". All other options have no effect. I attach a copy of the logfile in case it can be of help.KnimeBug.txt (309.5 KB)

When I open the table of the result with “Keep duplicate rows” and try to sort a column, it shows the same error.

Hope it helps

dasturiol · October 22, 2019, 2:28pm

Restarting KNIME fixes the problem until you try to repeat the action. So, it worked once and then I rerun the workflow and get the error.

Attach the logfileKnimeLog_afterReboot.txt (56.0 KB)

Further info. It seems there was a part of the workflow that did not run correctly and generated an empty cell in an SDF column. However, the task and workflow completed successfully.
After resetting that workflow and rerunning it, I do not have errors anymore.

swebb · March 3, 2020, 3:40pm

Did finding out how to trigger/fix this bug get anywhere? We still get the issue but haven’t yet worked out how to reproduce it.

Cheers

Sam

ChineS · April 2, 2020, 10:10am

So far this is the only available solution, thanks. Are there some news you are aware of?

Cheers,

Silvia

joshuahoran · April 24, 2020, 7:42pm

No updates from my end.

joshuahoran · February 24, 2021, 5:04pm

Hi All. After a couple years of additional experience with KNIME I think I have a better handle on these missing blob issues. I can confirm that these issues don’t seem to be tied a specific set of nodes (e.g. the MOE nodes as I had originally posted). Instead it appears that the Blob issues are a more general phenomena that arise after you try to upgrade the KNIME platform “in-place”. Upgrading extensions doesn’t trigger this, but updating KNIME itself does. Apparently things get messed up when KNIME tries to update itself (the old “repairing the plane while it it still flying” issue perhaps).

In any case, the solution is to never update KNIME “in-place” when a new version comes out. Instead separately download and install the new version and migrate your existing extensions over manually. So far this has worked every time.

dnaki · June 28, 2021, 4:50pm

Thanks for the tip, @joshuahoran !

I have also experienced the missing BLOB issue when running workflows with molecule column types (e.g. Mol2).
My workflows often involve long-running loops with molecule manipulations, so to make these workflows more robust I’ve resorted to converting molecule type columns to String columns right after they’re computed (I simply use the Java Snippet node where the molecule type column is represented as a String column type in the editor when selected from the column list).
When the Stringified molecule columns are needed again to perform some molecule task, I convert back to the molecule column type using the Molecule Type Cast node.
It’s a pain, and quite inelegant, but it increases the robustness of my workflows. Hopefully this BLOB bug can be squashed.

Note: I deal with large molecules (proteins), and a limited number of molecule column types, so I’m not sure whether all molecule types can be safely stringified and type cast.
Hope this helps someone. -Don