KNIME forgets tables

Hello,

With the latest releases of KNIME, several stability issues have arisen (yes I am a Mac user) and it often crashes. A new error is hat KNIME forgets its tables. All nodes in a workflow are green, but KNIME is not capable of displaying the data. The error obtained when tryong to work on these workflows is the following

 

ERROR Joiner Execute failed: Cannot read file "knime_container_20121005_7746834258226007474.bin.gz"

Reexcuting the whole workflow can be extremely time consuming as we sometime handle hundreds of mio lines. Any suggestions would be very welcome.

Thank you,

Marc

 

 

Hi Marc,

What do you mean by "crash"? Does the application close or does it freeze? We are aware of one problem in 2.6 that is going to be fixed in 2.7 (we'd like to fix it immediately but have given all node vendors time to review their nodes in case they have locally worked around it). You can enable this fix already by adding '-Dknime.core.dialog.edt=true' to your knime.ini file (requires 2.6 version).

The joiner problem: Can you have a look into the log file (<workspace>/.metadata/knime/knime.log) and see if it has more details (send us a call stack if available).

Also note the error message says: ERROR Joiner Execute failed -- so the node has not been completed execution (is not green?)

Bernd

Hi Bernd,

I see both kind of crashes: freezing up and quitting. I do tend to work with very large datasets with many millions of lines, so this might contribute to problems. 

The joiner was green and executed, but it had lost the data. When visualizing the table, I obtained a white table, not a grey one as when a node is not executed. I attach the log file.

I'll try the fix you propose, thank you for the tip.

bye,

Marc

Hi Marc,

Let me know what you find. The log file isn't all that interesting (no details on why the table files are missing).

Just to make sure we are on the same page: The problem you are seeing is with the standard KNIME Joiner node, not some specialized node from the HCS extension. I'm asking because I saw the below exception, which is in a "Join Layout" node contributed by the community HCS extension (which is also related to missing files). Maybe worth checking back with Antje (or whoever is in charge now in your team?)

  -- Bernd

2012-10-12 09:43:25,607 DEBUG ModalContext Join Layout : Loading model settings failed
java.lang.NullPointerException
	at java.io.File.<init>(File.java:222)
	at de.mpicbg.tds.core.ExcelLayout.openWorkbook(ExcelLayout.java:75)
	at de.mpicbg.tds.core.ExcelLayout.<init>(ExcelLayout.java:53)
	at de.mpicbg.tds.knime.hcstools.utils.JoinLayoutV2NodeModel.loadExcelSheet(JoinLayoutV2NodeModel.java:199)
	at de.mpicbg.tds.knime.hcstools.utils.JoinLayoutV2NodeModel.validateSettings(JoinLayoutV2NodeModel.java:235)
	at org.knime.core.node.Node.loadSettingsFrom(Node.java:553)
	at org.knime.core.node.Node.load(Node.java:317)
	at org.knime.core.node.NodePersistorVersion1xx.load(NodePersistorVersion1xx.java:688)
	at org.knime.core.node.workflow.SingleNodeContainerPersistorVersion1xx.loadNodeContainer(SingleNodeContainerPersistorVersion1xx.java:332)
	at org.knime.core.node.workflow.WorkflowManager.postLoad(WorkflowManager.java:6503)
	at org.knime.core.node.workflow.WorkflowManager.loadContent(WorkflowManager.java:6411)
	at org.knime.core.node.workflow.WorkflowManager.postLoad(WorkflowManager.java:6524)
	at org.knime.core.node.workflow.WorkflowManager.loadContent(WorkflowManager.java:6411)
	at org.knime.core.node.workflow.WorkflowManager.load(WorkflowManager.java:6308)
	at org.knime.core.node.workflow.WorkflowManager.load(WorkflowManager.java:6253)
	at org.knime.core.node.workflow.WorkflowManager.loadProject(WorkflowManager.java:6040)
	at org.knime.workbench.editor2.LoadWorkflowRunnable.run(LoadWorkflowRunnable.java:159)
	at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:121)

 

Hi Bernd,

we know about this error message but do not have a solution yet. It seems that the String of the SettingsModel for the file name can be 'null' though a file has been selected.

Whenever I try do debug this behaviour it disappears for some reason and is not reproducible anymore. I'm wondering whether it has something to do with the default component, which is defined like this:

addDialogComponent(new DialogComponentFileChooser(fileModelSetting, "plateLayoutFile", JFileChooser.OPEN_DIALOG, ".xls|.xlsx"));

Anyhow, I'll try to define the allowed file types now like this: ".xls",".xlsx"

We''ll see whether this helps. If you have any suggestions, let me know.

Marcs problem is not related to this problem and we have seen it with standard KNIME nodes (not necessarily the Joiner). My impression is that KNIME looses data at some point (if it's running longer time and handling big data). What kind of file it this "knime_container_*.bin.gz"? Where is it created and when?

How "long" is long-running in your cases? I remember one user which had his workflow open for several days and then a cron job removed all files in /tmp that were "too old". The knime_container_*.bin.gz file are temporary copies of node outputs and if you delete them a node does not finds its output any more. The temporary files get written into the workflow directory only when the workflow is saved.

This is what I have guessed. Maybe we should find out how long a Mac keeps its temp-files and if this can be modified.

This brings us back to the autosave-feature request... :-)