Help with Unresponsive Table Output view.

Hi KNIME Devs!

I have written a node which reads many data files, each into it's own row in a custom data cell type. I am doing some stress testing on it now using a dataset containing ~200 100mb files (about 600 million doubles total). This is a heavy but realistic scenario.

While I am working on a real renderer for this cell type, I am using a string renderer with the filename of the original file as the datacell renderer. When opening the executed output of the node, I notice that the table window takes a about 2 minutes to render and that it renders the first 50 rows.  Scrolling down the table also throws an out of memory exception shown below when I get to about the 70th row.  This would be consistent with the table completely de-serializing all of the data for 50 rows and then additionally serializing more rows into memory until it exceeds available heap space.  

Does this sound like expected behavior from your point of view? Is there anything I can do to work around this?

 

Thanks,

Aaron

 

Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
    at io.landysh.inflor.java.core.ColumnStore.load(ColumnStore.java:187)
    at io.landysh.inflor.java.knime.dataTypes.columnStoreCell.ColumnStoreCell$ColumnStoreCellSerializer.deserialize(ColumnStoreCell.java:32)
    at io.landysh.inflor.java.knime.dataTypes.columnStoreCell.ColumnStoreCell$ColumnStoreCellSerializer.deserialize(ColumnStoreCell.java:1)
    at org.knime.core.data.container.DCObjectInputVersion2.readDataCellPerKNIMESerializer(DCObjectInputVersion2.java:114)
    at org.knime.core.data.container.BufferFromFileIteratorVersion20$DataCellStreamReader.readDataCell(BufferFromFileIteratorVersion20.java:339)
    at org.knime.core.data.container.BufferFromFileIteratorVersion20.next(BufferFromFileIteratorVersion20.java:180)
    at org.knime.core.data.container.Buffer$FromFileIterator.next(Buffer.java:1)
    at org.knime.core.node.BufferedDataTable$CloseableFailProveRowIterator.next(BufferedDataTable.java:420)
    at org.knime.core.node.tableview.TableContentModel.cacheNextRow(TableContentModel.java:1129)
    at org.knime.core.node.tableview.TableContentModel.getRow(TableContentModel.java:1094)
    at org.knime.core.node.tableview.TableContentModel.isHiLit(TableContentModel.java:918)
    at org.knime.core.node.tableview.TableContentView.prepareRenderer(TableContentView.java:283)
    at javax.swing.plaf.basic.BasicTableUI.paintCell(BasicTableUI.java:2114)
    at javax.swing.plaf.basic.BasicTableUI.paintCells(BasicTableUI.java:2016)
    at javax.swing.plaf.basic.BasicTableUI.paint(BasicTableUI.java:1812)
    at javax.swing.plaf.ComponentUI.update(ComponentUI.java:161)
    at javax.swing.JComponent.paintComponent(JComponent.java:780)
    at javax.swing.JComponent.paint(JComponent.java:1056)
    at javax.swing.JComponent.paintToOffscreen(JComponent.java:5210)
    at javax.swing.RepaintManager$PaintManager.paintDoubleBuffered(RepaintManager.java:1579)
    at javax.swing.RepaintManager$PaintManager.paint(RepaintManager.java:1502)
    at javax.swing.RepaintManager.paint(RepaintManager.java:1272)
    at javax.swing.JComponent.paintForceDoubleBuffered(JComponent.java:1092)
    at javax.swing.JViewport.blitDoubleBuffered(JViewport.java:1605)
    at javax.swing.JViewport.windowBlitPaint(JViewport.java:1558)
    at javax.swing.JViewport.setViewPosition(JViewport.java:1100)
    at javax.swing.plaf.basic.BasicScrollPaneUI$Handler.vsbStateChanged(BasicScrollPaneUI.java:1060)
    at javax.swing.plaf.basic.BasicScrollPaneUI$Handler.stateChanged(BasicScrollPaneUI.java:1049)
    at javax.swing.DefaultBoundedRangeModel.fireStateChanged(DefaultBoundedRangeModel.java:364)
    at javax.swing.DefaultBoundedRangeModel.setRangeProperties(DefaultBoundedRangeModel.java:302)
    at javax.swing.DefaultBoundedRangeModel.setValue(DefaultBoundedRangeModel.java:168)
    at javax.swing.JScrollBar.setValue(JScrollBar.java:477)

Hi Aaron,

~200 100mb files (about 600 million doubles total).

Does 1 file correspond to one row (= 3 Mio columns)?  I guess not. So it's probably one column only that comprises a very long vector? Is that your custom cell type that is backed by a file store?

The table view uses a sliding window over the data. While you scroll down it will keep something like 500 rows in memory.

- Bernd