Hi KNIME Devs!
I have written a node which reads many data files, each into it's own row in a custom data cell type. I am doing some stress testing on it now using a dataset containing ~200 100mb files (about 600 million doubles total). This is a heavy but realistic scenario.
While I am working on a real renderer for this cell type, I am using a string renderer with the filename of the original file as the datacell renderer. When opening the executed output of the node, I notice that the table window takes a about 2 minutes to render and that it renders the first 50 rows. Scrolling down the table also throws an out of memory exception shown below when I get to about the 70th row. This would be consistent with the table completely de-serializing all of the data for 50 rows and then additionally serializing more rows into memory until it exceeds available heap space.
Does this sound like expected behavior from your point of view? Is there anything I can do to work around this?
Thanks,
Aaron
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
at io.landysh.inflor.java.core.ColumnStore.load(ColumnStore.java:187)
at io.landysh.inflor.java.knime.dataTypes.columnStoreCell.ColumnStoreCell$ColumnStoreCellSerializer.deserialize(ColumnStoreCell.java:32)
at io.landysh.inflor.java.knime.dataTypes.columnStoreCell.ColumnStoreCell$ColumnStoreCellSerializer.deserialize(ColumnStoreCell.java:1)
at org.knime.core.data.container.DCObjectInputVersion2.readDataCellPerKNIMESerializer(DCObjectInputVersion2.java:114)
at org.knime.core.data.container.BufferFromFileIteratorVersion20$DataCellStreamReader.readDataCell(BufferFromFileIteratorVersion20.java:339)
at org.knime.core.data.container.BufferFromFileIteratorVersion20.next(BufferFromFileIteratorVersion20.java:180)
at org.knime.core.data.container.Buffer$FromFileIterator.next(Buffer.java:1)
at org.knime.core.node.BufferedDataTable$CloseableFailProveRowIterator.next(BufferedDataTable.java:420)
at org.knime.core.node.tableview.TableContentModel.cacheNextRow(TableContentModel.java:1129)
at org.knime.core.node.tableview.TableContentModel.getRow(TableContentModel.java:1094)
at org.knime.core.node.tableview.TableContentModel.isHiLit(TableContentModel.java:918)
at org.knime.core.node.tableview.TableContentView.prepareRenderer(TableContentView.java:283)
at javax.swing.plaf.basic.BasicTableUI.paintCell(BasicTableUI.java:2114)
at javax.swing.plaf.basic.BasicTableUI.paintCells(BasicTableUI.java:2016)
at javax.swing.plaf.basic.BasicTableUI.paint(BasicTableUI.java:1812)
at javax.swing.plaf.ComponentUI.update(ComponentUI.java:161)
at javax.swing.JComponent.paintComponent(JComponent.java:780)
at javax.swing.JComponent.paint(JComponent.java:1056)
at javax.swing.JComponent.paintToOffscreen(JComponent.java:5210)
at javax.swing.RepaintManager$PaintManager.paintDoubleBuffered(RepaintManager.java:1579)
at javax.swing.RepaintManager$PaintManager.paint(RepaintManager.java:1502)
at javax.swing.RepaintManager.paint(RepaintManager.java:1272)
at javax.swing.JComponent.paintForceDoubleBuffered(JComponent.java:1092)
at javax.swing.JViewport.blitDoubleBuffered(JViewport.java:1605)
at javax.swing.JViewport.windowBlitPaint(JViewport.java:1558)
at javax.swing.JViewport.setViewPosition(JViewport.java:1100)
at javax.swing.plaf.basic.BasicScrollPaneUI$Handler.vsbStateChanged(BasicScrollPaneUI.java:1060)
at javax.swing.plaf.basic.BasicScrollPaneUI$Handler.stateChanged(BasicScrollPaneUI.java:1049)
at javax.swing.DefaultBoundedRangeModel.fireStateChanged(DefaultBoundedRangeModel.java:364)
at javax.swing.DefaultBoundedRangeModel.setRangeProperties(DefaultBoundedRangeModel.java:302)
at javax.swing.DefaultBoundedRangeModel.setValue(DefaultBoundedRangeModel.java:168)
at javax.swing.JScrollBar.setValue(JScrollBar.java:477)