Database Reader : Java Heap Space Error on Configure

Jay · August 26, 2008, 6:42pm

Hi,

When I go to configure (an unexecuted) database reader node I get a FATAL configure failed message along with a Java Heap Space message.

Does the node somehow test the potential size of the table? Any ideas?

INFO 	 WorkflowEditor	 Setting console log level to DEBUG
DEBUG	 WorkflowContextMenuProvider	 Building up context menu...
DEBUG	 WorkflowContextMenuProvider	 adding open node-view action(s) to context menu...
DEBUG	 WorkflowContextMenuProvider	 adding open port-view action(s) to context menu...
DEBUG	 OpenDialogAction	 Opening node dialog...
INFO 	 Database Reader	 reset
FATAL	 Database Reader	 Configure failed
FATAL	 Database Reader	 Java heap space
DEBUG	 Database Reader	 Configure failed
java.lang.OutOfMemoryError: Java heap space
INFO 	 Database Reader	 reset
DEBUG	 WorkflowEditor	 Workflow event triggered: WorkflowEvent [type=NodeReset;old=null;new=null;timestamp=Aug 26, 2008 12:38:32 PM]
DEBUG	 WorkflowRootEditPart	 WorkflowRoot: workflow changed, refreshing children/connections..

Many thanks,

Jay

unknown_user · August 29, 2008, 11:29am

Hi Jay,
Interesting problem - can you reproduce it? Can you please send me the complete stack trace? I have had a look into the code and could not detect and potential memory leak. During configure, I do not test any size of the table, neither allocate any memory to later fill the table. Only the SQL query is execute to retrieve the database meta data; the statement is then released and re-initiated during execute where we only allow access via an iterator to the database. By any chance, did you execute the Pivot node at the same time? :wink:
Cheers, Thomas

Jay · April 8, 2009, 3:18pm

Hi Thomas,

I’m experiencing this issue again where the database reader node will fail to configure but shrinking the size of the table, through a WHERE clause, allows the node to work.

This is not a crippling error but it’s a bit cumbersome when you end up having to do multiple loads.

Also somewhat unrelated but I’m finding that the new Knime leaves more around in RAM if nodes failed or after a workflow has been closed simply from general usage. Sometimes in the area of several hundred megabytes. Any ideas?

Many thanks,

Jay

gabriel · April 8, 2009, 3:55pm

Hi Jay,
By any chance, are you using a mySQL database? We have encountered a similar problem in this database driver, which seems to ignore the fetch size while reading ALL data into the ResultSet. If this heap space error is thrown, you should be able to see the complete stack trace in the console log (working in DEBUG mode only). Can you please provide us with this information.
Cheers, Thomas

Jay · April 8, 2009, 4:40pm

Hi Gabriel,

I am using a local MySQL db. Thanks for your rapid reply.

DEBUG NodeContainerEditPart Database Reader 0:2:519 (CONFIGURED)
DEBUG WorkflowContextMenuProvider Building up context menu…
DEBUG WorkflowContextMenuProvider selected edit part: NodeContainerEditPart( Database Reader 0:2:519 (CONFIGURED) )
DEBUG WorkflowContextMenuProvider adding open node-view action(s) to context menu…
DEBUG WorkflowContextMenuProvider adding open port-view action(s) to context menu…
DEBUG ExecuteAction Creating execution job for 1 node(s)…
DEBUG NodeContainer Database Reader 0:2:519 has new state: MARKEDFOREXEC
DEBUG NodeContainer Database Reader 0:2:519 has new state: QUEUED
DEBUG KnimeResourceNavigator state changed to EXECUTING
DEBUG NodeContainer Workflow Manager 0:2 has new state: EXECUTING
DEBUG NodeContainer Workflow Manager 0 has new state: EXECUTING
DEBUG WorkflowManager Database Reader 0:2:519 doBeforeExecute
DEBUG NodeContainer Database Reader 0:2:519 has new state: EXECUTING
DEBUG NodeContainer Workflow Manager 0:2 has new state: EXECUTING
DEBUG Database Reader Start execute
DEBUG Database Reader reset
DEBUG Database Reader clean output ports.
ERROR Database Reader Execute failed: java.lang.OutOfMemoryError: Java heap space
DEBUG Database Reader Execute failed: java.lang.OutOfMemoryError: Java heap space
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
at org.knime.base.node.io.database.DBReaderNodeModel.execute(DBReaderNodeModel.java:87)
at org.knime.core.node.NodeModel.execute(NodeModel.java:556)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:410)
at org.knime.core.node.Node.execute(Node.java:653)
at org.knime.core.node.workflow.SingleNodeContainer.executeNode(SingleNodeContainer.java:587)
at org.knime.core.node.workflow.SingleNodeContainer.access$1(SingleNodeContainer.java:561)
at org.knime.core.node.workflow.SingleNodeContainer$1.run(SingleNodeContainer.java:446)
at org.knime.core.node.workflow.JobRunnable.run(JobRunnable.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:98)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:166)
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1574)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1398)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
at org.knime.core.node.port.database.DatabaseReaderConnection.createTable(DatabaseReaderConnection.java:173)
at org.knime.base.node.io.database.DBReaderNodeModel.execute(DBReaderNodeModel.java:79)
… 12 more
DEBUG Database Reader reset
DEBUG Database Reader clean output ports.
DEBUG Database Reader Configure succeeded. (Database Reader)
DEBUG WorkflowManager Database Reader 0:2:519 doAfterExecute - failure
DEBUG NodeContainer Database Reader 0:2:519 has new state: CONFIGURED
DEBUG Column Combiner Configure succeeded. (Column Combiner)
DEBUG NodeContainer Column Combiner 0:2:3 has new state: CONFIGURED
DEBUG Joiner Configure succeeded. (Joiner)
DEBUG NodeContainer Joiner 0:2:6 has new state: CONFIGURED
DEBUG Column Filter Configure succeeded. (Column Filter)
DEBUG NodeContainer Column Filter 0:2:24 has new state: CONFIGURED
DEBUG Row Filter Configure succeeded. (Row Filter)
DEBUG NodeContainer Row Filter 0:2:45 has new state: CONFIGURED
DEBUG Row Filter Configure succeeded. (Row Filter)
DEBUG NodeContainer Row Filter 0:2:46 has new state: CONFIGURED
DEBUG Java Snippet Configure succeeded. (Java Snippet)
DEBUG NodeContainer Java Snippet 0:2:688 has new state: CONFIGURED
DEBUG Row Filter Configure succeeded. (Row Filter)
DEBUG NodeContainer Row Filter 0:2:687 has new state: CONFIGURED
DEBUG Column Filter Configure succeeded. (Column Filter)
DEBUG NodeContainer Column Filter 0:2:689 has new state: CONFIGURED
DEBUG Sorter Configure succeeded. (Sorter)
DEBUG NodeContainer Sorter 0:2:27 has new state: CONFIGURED
DEBUG Java Snippet Configure succeeded. (Java Snippet)
DEBUG NodeContainer Java Snippet 0:2:28 has new state: CONFIGURED
DEBUG RowID Configure succeeded. (RowID)
DEBUG NodeContainer RowID 0:2:29 has new state: CONFIGURED
DEBUG Java Snippet Configure succeeded. (Java Snippet)
DEBUG NodeContainer Java Snippet 0:2:30 has new state: CONFIGURED
DEBUG RowID Configure succeeded. (RowID)
DEBUG NodeContainer RowID 0:2:32 has new state: CONFIGURED
DEBUG Column Filter Configure succeeded. (Column Filter)
DEBUG NodeContainer Column Filter 0:2:35 has new state: CONFIGURED
DEBUG Rename Configure succeeded. (Rename)
DEBUG NodeContainer Rename 0:2:36 has new state: CONFIGURED
DEBUG Joiner Configure succeeded. (Joiner)
DEBUG NodeContainer Joiner 0:2:38 has new state: CONFIGURED
DEBUG Java Snippet Configure succeeded. (Java Snippet)
DEBUG NodeContainer Java Snippet 0:2:37 has new state: CONFIGURED
DEBUG Row Filter Configure succeeded. (Row Filter)
DEBUG NodeContainer Row Filter 0:2:39 has new state: CONFIGURED
DEBUG Java Snippet Configure succeeded. (Java Snippet)
DEBUG NodeContainer Java Snippet 0:2:530 has new state: CONFIGURED
DEBUG Row Filter Configure succeeded. (Row Filter)
DEBUG NodeContainer Row Filter 0:2:108 has new state: CONFIGURED
DEBUG String To Number Configure succeeded. (String To Number)
DEBUG NodeContainer String To Number 0:2:686 has new state: CONFIGURED
DEBUG Column Filter Configure succeeded. (Column Filter)
DEBUG NodeContainer Column Filter 0:2:216 has new state: CONFIGURED
DEBUG Rule Engine Configure succeeded. (Rule Engine)
DEBUG NodeContainer Rule Engine 0:2:706 has new state: CONFIGURED
WARN Nominal Value Row Filter No nominal columns with possible values found! Execute predecessor or check input table.
DEBUG NodeContainer Nominal Value Row Filter 0:2:707 has new state: IDLE
DEBUG KnimeResourceNavigator state changed to IDLE
DEBUG NodeContainer Workflow Manager 0:2 has new state: IDLE
DEBUG NodeContainer Workflow Manager 0 has new state: IDLE
DEBUG NodeContainer Workflow Manager 0:2 has new state: IDLE

gabriel · April 8, 2009, 5:23pm

Hi Jay,
This is exactly the same error we have had. In the lines after “Caused By”, you see that this issues is somehow driver related. Sorry, I currently don’t have a work-around for you, but I have created a bug entry in order to fix this in one of our next release.
Best, Thomas

Jay · April 8, 2009, 5:56pm

Hi,

Okay, thanks for taking a look for me. That was a very fast turn around.

All the best,

Jay