Size limit of input tables for R Snippet

Hi,

I have problems loading data tables above a certain size into R Snippets.

For a 2-column table I'm using, the R Snippet fails if the input table has more than 19900 rows (code is simply 'rOut <- kIn', example workflow attached).  For a 1-column table, I get different maximum number of input rows depending on column data type.

The error msg (in console) is: "Execute failed: Failed to convert table node inputs into r workspace variables: org.rosuda.REngine.Rserve.RserveException: eval failed, request status: error code: 127".

I'm using KNIME 2.11.2, R Scripting extension 2.0.2.201502181300.

Thanks in advance for any help,

Andrew

Console msg at DEBUG level:

 

INFO      KNIMECorePlugin                    Setting console view log level to DEBUG
DEBUG     NodeContainerEditPart              R Snippet 0:16 (CONFIGURED)
DEBUG     ExecuteAction                      Creating execution job for 1 node(s)...
DEBUG     NodeContainer                      Setting dirty flag on R Snippet 0:16
DEBUG     NodeContainer                      Setting dirty flag on moo 0
DEBUG     NodeContainer                      R Snippet 0:16 has new state: CONFIGURED_MARKEDFOREXEC
DEBUG     NodeContainer                      R Snippet 0:16 has new state: CONFIGURED_QUEUED
DEBUG     WorkflowEditor                     Workflow event triggered: WorkflowEvent [type=WORKFLOW_DIRTY;node=0;old=null;new=null;timestamp=Mar 13, 2015 2:32:15 PM]
DEBUG     WorkflowRootEditPart               WorkflowRoot: workflow changed, refreshing children/connections..
DEBUG     NodeContainer                      moo 0 has new state: EXECUTING
DEBUG     WorkflowManager                    R Snippet 0:16 doBeforePreExecution
DEBUG     NodeContainer                      ROOT  has new state: EXECUTING
DEBUG     NodeContainer                      R Snippet 0:16 has new state: PREEXECUTE
DEBUG     WorkflowManager                    R Snippet 0:16 doBeforeExecution
DEBUG     NodeContainer                      R Snippet 0:16 has new state: EXECUTING
DEBUG     WorkflowFileStoreHandlerRepository     Adding handler 73029838-4615-4d5a-9c6a-15dc6765abea (R Snippet 0:16: <no directory>) - 10 in total
DEBUG     LocalNodeExecutionJob              R Snippet 0:16 Start execute
DEBUG     R Snippet                          reset
ERROR     R Snippet                          Execute failed: Failed to convert table node inputs into r workspace variables: org.rosuda.REngine.Rserve.RserveException: eval failed, request status: error code: 127
DEBUG     R Snippet                          Execute failed: Failed to convert table node inputs into r workspace variables: org.rosuda.REngine.Rserve.RserveException: eval failed, request status: error code: 127
de.mpicbg.knime.scripting.core.exceptions.KnimeScriptingException: Failed to convert table node inputs into r workspace variables: org.rosuda.REngine.Rserve.RserveException: eval failed, request status: error code: 127
    at de.mpicbg.knime.scripting.r.RUtils.pushToR(RUtils.java:640)
    at de.mpicbg.knime.scripting.r.RSnippetNodeModel.execute(RSnippetNodeModel.java:53)
    at org.knime.core.node.NodeModel.execute(NodeModel.java:706)
    at org.knime.core.node.NodeModel.executeModel(NodeModel.java:555)
    at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1131)
    at org.knime.core.node.Node.execute(Node.java:927)
    at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
    at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
    at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
    at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
    at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
    at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:125)
    at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:248)
DEBUG     WorkflowManager                    R Snippet 0:16 doBeforePostExecution
DEBUG     NodeContainer                      R Snippet 0:16 has new state: POSTEXECUTE
DEBUG     WorkflowManager                    R Snippet 0:16 doAfterExecute - failure
DEBUG     R Snippet                          reset
DEBUG     R Snippet                          clean output ports.
DEBUG     WorkflowFileStoreHandlerRepository     Removing handler 73029838-4615-4d5a-9c6a-15dc6765abea (R Snippet 0:16: <no directory>) - 9 remaining
DEBUG     NodeContainer                      R Snippet 0:16 has new state: IDLE
DEBUG     R Snippet                          Configure succeeded. (R Snippet)
DEBUG     NodeContainer                      R Snippet 0:16 has new state: CONFIGURED
DEBUG     NodeContainer                      moo 0 has new state: CONFIGURED
DEBUG     NodeContainer                      ROOT  has new state: IDLE

 

Thanks for reporting the problem.

Yes, there is a size limit. Though in my case it is different. Currently, I try to find a solution that the transfer can be splitted up into pieces.

One thing you can try:

Check the Rserve configuration settings with the following command(s)

Console:

R CMD Rserve --RS-settings

Within R:

Rserve(args = "--RS-settings")

This should display the location fo your configuration file and the size limit ("max.input buffer size") like this:

Rserve v1.7-3

config file: /etc/Rserv.conf
working root: /tmp/Rserv
port: 6311
local socket: [none, TCP/IP used]
authorization required: no
plain text password: not allowed
passwords file: [none]
allow I/O: yes
allow remote access: yes
control commands: no
interactive: yes
max.input buffer size: 500000 kB

Try to increase that value or even try to delete it (it is there by default but as far as I remember it is save to delete that configuration).

Could you please let me know if that solves your problem?

Thanks!

Antje
 

If the config file does not exist (though it claims to have one at /etc/... ), you might want to create a new one. Either at this default location, or somewhere else.

Then you need to start Rserve with the argument for config file location

--RS-conf <file>

 

Hi Antje,

Thanks a lot, that does indeed fix the issue.

I should have stated in the original post that this problem only arose after upgrading to the most recent version of KNIME and the R Snippets.  On older installations this was not an issue.  (unfortunately, I can't say which was the most recent version where it worked, because I no longer have access to that computer, but I'm fairly certain it was least 2.10.  I just tested it and it worked for 2.9).

Thanks again for your help,

 

Andrew

Hi Andrew,

the size limit was always an issue. One thing which changed is the transfer of the row IDs to R. Of course, this is additional data... This might explain the behaviour of the different versions.

Antje

Greetings -

I'm brand new to KNIME and very interested in the R interactive node functionality.  I have a table that is approx. 90,000 rows by 1500 cols.  I connect the successfully executed Table Reader node's output port to an R Snippet input port.  The R Snippet node recognizes the table's column names.  But when I try to run any code on the knime.in object, from within the Configuration, everything freezes up.  Similarly, when I close the Configuration window, and try to Execute the R Snippet node, it starts but progresses very slowly.  Then ultimately I get the error "red x" symbol on the node icon, and a mouseover reads "Execute failed: GC overhead limit exceeded."

Please Advise?

Thanks!

packjt1

Hi packjt1,

as mentioned above the size of the table was always an issue. The latest version of the Scripting Integration (Snippet adn Plot node) has an additional tab "Chunk Settings" which is per default on -1, meaning off (all data transferred in one chunk). You can specify the chucks sizes (colums for pushing data and rows for pulling data) for the data transfer (see also help).

In R the chunks are automatically put together again as one table, so nothing to worry about. Try to use smaller chucks (reduced the transfer speed a little bit) and see if the problem is solved.

Best,

Martin

 

Martin - Great - I will try this.  Thanks so much for your feedback.

Josh