Intermittent sorting error

swebb · May 3, 2016, 4:41pm

G'afternoon

A couple of my colleagues are experiancing some sorting issues. I've tried to recreate the issue on my machine but failed so apologies for the partially lacking bug report.

Bug seen in the presence of RDKit nodes. On one machine it could be triggered by providing a invalid SMARTS to the Substructure filter node. However, this didn't occur for another user who had the same issue.

I've attached the log showing various exceptions being thrown such as:

2016-05-03 13:58:02,270 : DEBUG : SwingWorker-pool-6-thread-3 : TableSorterWorker : : : Starting interactive table sorting on column(s) [SMARTS]
2016-05-03 13:58:02,271 : DEBUG : SwingWorker-pool-6-thread-3 : Buffer : : : Opening input stream on file "C:\Users\davidp\AppData\Local\Temp\knime_MSSDB47021\knime_container_20160503_3195580412025055733.bin.gz", 2 open streams
2016-05-03 13:58:02,273 : DEBUG : SwingWorker-pool-6-thread-3 : Buffer : : : Opening input stream on file "C:\Users\davidp\AppData\Local\Temp\knime_MSSDB47021\knime_container_20160503_5073153588267695046.bin.gz", 2 open streams
2016-05-03 13:58:02,274 : DEBUG : SwingWorker-pool-6-thread-3 : Buffer : : : Opening input stream on file "C:\Users\davidp\AppData\Local\Temp\knime_MSSDB47021\knime_container_20160503_1876006222211227932.bin.gz", 2 open streams
2016-05-03 13:58:02,333 : DEBUG : SwingWorker-pool-6-thread-3 : Buffer : : : Closing input stream on "C:\Users\davidp\AppData\Local\Temp\knime_MSSDB47021\knime_container_20160503_1876006222211227932.bin.gz", 1 remaining
2016-05-03 13:58:02,351 : ERROR : AWT-EventQueue-0 : TableSorterWorker : : : Error sorting table: Error while writing to buffer: Blob file location "C:\Program Files\KNIME\col_0\000\000" does not exist
2016-05-03 13:58:02,352 : DEBUG : AWT-EventQueue-0 : TableSorterWorker : : : Error sorting table: Error while writing to buffer: Blob file location "C:\Program Files\KNIME\col_0\000\000" does not exist
java.util.concurrent.ExecutionException: org.knime.core.data.container.DataContainerException: Error while writing to buffer: Blob file location "C:\Program Files\KNIME\col_0\000\000" does not exist
   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
   at java.util.concurrent.FutureTask.get(FutureTask.java:192)
   at javax.swing.SwingWorker.get(SwingWorker.java:602)
   at org.knime.core.node.tableview.TableSorterWorker.done(TableSorterWorker.java:225)
   at javax.swing.SwingWorker$5.run(SwingWorker.java:737)
   at javax.swing.SwingWorker$DoSubmitAccumulativeRunnable.run(SwingWorker.java:832)
   at sun.swing.AccumulativeRunnable.run(AccumulativeRunnable.java:112)
   at javax.swing.SwingWorker$DoSubmitAccumulativeRunnable.actionPerformed(SwingWorker.java:842)
   at javax.swing.Timer.fireActionPerformed(Timer.java:313)
   at javax.swing.Timer$DoPostEvent.run(Timer.java:245)
   at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
   at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756)
   at java.awt.EventQueue.access$500(EventQueue.java:97)
   at java.awt.EventQueue$3.run(EventQueue.java:709)
   at java.awt.EventQueue$3.run(EventQueue.java:703)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
   at java.awt.EventQueue.dispatchEvent(EventQueue.java:726)
   at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
   at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
   at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:109)
   at java.awt.WaitDispatchSupport$2.run(WaitDispatchSupport.java:184)
   at java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:229)
   at java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:227)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.awt.WaitDispatchSupport.enter(WaitDispatchSupport.java:227)
   at java.awt.Dialog.show(Dialog.java:1084)
   at java.awt.Component.show(Component.java:1673)
   at java.awt.Component.setVisible(Component.java:1625)
   at java.awt.Window.setVisible(Window.java:1014)
   at java.awt.Dialog.setVisible(Dialog.java:1005)
   at org.knime.core.node.tableview.TableSorterWorker$1.run(TableSorterWorker.java:145)
   at org.knime.core.node.util.ViewUtils.invokeAndWaitInEDT(ViewUtils.java:134)
   at org.knime.core.node.tableview.TableSorterWorker.executeAndShowProgress(TableSorterWorker.java:141)
   at org.knime.core.node.tableview.TableContentModel.sortTableIntern(TableContentModel.java:615)
   at org.knime.core.node.tableview.TableContentModel.access$2(TableContentModel.java:595)
   at org.knime.core.node.tableview.TableContentModel$2.run(TableContentModel.java:587)
   at org.knime.core.node.util.ViewUtils.invokeAndWaitInEDT(ViewUtils.java:134)
   at org.knime.core.node.tableview.TableContentModel.requestSort(TableContentModel.java:584)
   at org.knime.core.node.tableview.TableContentView$1SortKeyActionListener.actionPerformed(TableContentView.java:794)
   at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
   at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
   at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
   at javax.swing.JToggleButton$ToggleButtonModel.setPressed(JToggleButton.java:308)
   at javax.swing.AbstractButton.doClick(AbstractButton.java:376)
   at javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:833)
   at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(BasicMenuItemUI.java:877)
   at java.awt.Component.processMouseEvent(Component.java:6535)
   at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
   at java.awt.Component.processEvent(Component.java:6300)
   at java.awt.Container.processEvent(Container.java:2236)
   at java.awt.Component.dispatchEventImpl(Component.java:4891)
   at java.awt.Container.dispatchEventImpl(Container.java:2294)
   at java.awt.Component.dispatchEvent(Component.java:4713)
   at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
   at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4525)
   at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
   at java.awt.Container.dispatchEventImpl(Container.java:2280)
   at java.awt.Window.dispatchEventImpl(Window.java:2750)
   at java.awt.Component.dispatchEvent(Component.java:4713)
   at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
   at java.awt.EventQueue.access$500(EventQueue.java:97)
   at java.awt.EventQueue$3.run(EventQueue.java:709)
   at java.awt.EventQueue$3.run(EventQueue.java:703)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
   at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
   at java.awt.EventQueue$4.run(EventQueue.java:731)
   at java.awt.EventQueue$4.run(EventQueue.java:729)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
   at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
   at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
   at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
   at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
   at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
   at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
   at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: org.knime.core.data.container.DataContainerException: Error while writing to buffer: Blob file location "C:\Program Files\KNIME\col_0\000\000" does not exist
   at org.knime.core.data.container.DataContainer.checkAsyncWriteThrowable(DataContainer.java:517)
   at org.knime.core.data.container.DataContainer.offerToAsynchronousQueue(DataContainer.java:664)
   at org.knime.core.data.container.DataContainer.addRowToTable(DataContainer.java:785)
   at org.knime.core.data.sort.AbstractTableSorter.sortOnDisk(AbstractTableSorter.java:521)
   at org.knime.core.data.sort.AbstractTableSorter.sortInternal(AbstractTableSorter.java:423)
   at org.knime.core.data.sort.DataTableSorter.sort(DataTableSorter.java:213)
   at org.knime.core.node.tableview.TableSorterWorker.doInBackground(TableSorterWorker.java:191)
   at org.knime.core.node.tableview.TableSorterWorker.doInBackground(TableSorterWorker.java:1)
   at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at javax.swing.SwingWorker.run(SwingWorker.java:334)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error while writing to buffer: Blob file location "C:\Program Files\KNIME\col_0\000\000" does not exist
   at org.knime.core.data.container.Buffer.addRow(Buffer.java:636)
   at org.knime.core.data.container.DataContainer.addRowToTableWrite(DataContainer.java:500)
   at org.knime.core.data.container.DataContainer.access$6(DataContainer.java:461)
   at org.knime.core.data.container.DataContainer$ASyncWriteCallable.callWithContext(DataContainer.java:1244)
   at org.knime.core.data.container.DataContainer$ASyncWriteCallable.call(DataContainer.java:1214)
   at org.knime.core.data.container.DataContainer$ASyncWriteCallable.call(DataContainer.java:1)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   ... 3 more
Caused by: java.io.IOException: Blob file location "C:\Program Files\KNIME\col_0\000\000" does not exist
   at org.knime.core.data.container.Buffer.getBlobFile(Buffer.java:1624)
   at org.knime.core.data.container.Buffer.handleIncomingBlob(Buffer.java:848)
   at org.knime.core.data.container.Buffer.handleIncomingBlob(Buffer.java:738)
   at org.knime.core.data.container.Buffer.saveBlobsAndFileStores(Buffer.java:685)
   at org.knime.core.data.container.Buffer.addRow(Buffer.java:609)
   ... 9 more

After updating to the latest KNIME the following actions caused the bug (on the first machine, works fine on mine)

1) run workflow until node in RDKIt Substructure filter throws exception

2) reset loop

3) run first iteration to column resorter

4) View output table of column resorter

5) Attempt to sort

Is this something that's been seen before? I'm not sure why "C:\Program Files\KNIME\col_0\000\000" would be chosen as file path. I'm running from a extracted zip, my colleague used the installer.

Cheers

Sam

wiswedel · May 19, 2016, 9:43am

Hi Sam,

I know this won't help much but maybe I can clarify some points to the exten that I understand them. First of all I believe this is an old problem because that piece of code hasn't changed for years. It must be related to blobs (used by RDKit), loops, and some usage scenario with partially executing loops, resets, etc. Being able to reproduce it is crucial for us to understand (and then fix) it.

Blobs are "larger" cells that are saved in their own file somewhere in the temp folder. The path from your error message (C:\Program Files\KNIME\col_0\000\000") seems wrong. "col_0\000\000" is the path relative to the table's temporary storage (it's some folder hierarchy as it avoids to have potentially millions of files in a single directory). The fact that this is directly in "program files/knime" indicates that the table space is corrupt/gone (dunno why).

I suppose this doesn't happen anymore when you reset the entire workflow and then start over from scratch?

Bernd

swebb · May 19, 2016, 11:51am

Hi again

> I suppose this doesn't happen anymore when you reset the entire workflow and then start over from scratch?

The whole workflow didn't need to be reset I dont think, just a certain distance back. I will try to work with the people who had the issue and see whether I can get you an example workflow with some dummy data.

Cheers

Sam