New R integration for 3.1

Hi,

I assume this information will eventually come out, but can the community get more information about the recent changes to KNIME 3.1 and R? I guess Rserve has replaced rJava, and the reason was so that multiple R nodes could be run in parallel? Do we no longer need rJava and to play with the knime.ini file to get everything working? Will the structural differences between using Rserve and rJava have any obvious effects we should be aware of? Anything else? 

Any details would be appreciated. Thanks.

-Brian Muchmore

Hi Brian,

Yes, we changed the library that enables communication between KNIME and R. Previoulsly this was rJava and is now RServe. This was motivated by some shortcommings of the rJava based integration: this was mostly around global locking preventing parallel execution of independent R nodes  and some scalability problems (like R AND KNIME crashing if you feed it with too many data; or the entire data copied twice to disk to get it from one system to the other). We've also been in contact with the developers owning the rJava and RServe packages at the R project. They motivated us using RServe.

In theory that wouldn't have been a significant change as both of these libraries follow the same interface (REngine), except for some trivial Rserve book keeping of instances to start and shutdown. Unfortunately that's only the theory as we learn now that 3.1.0 is out. We are aware of the following problems:

  • Bad error message in case RServe is not installed into R (not a problem on most systems, specifically Windows as 'our' R binaries contain this package). Clearly a bug and will be fixed soon.
  • The problems around device printing as discussed here. Even though they seem to conclude it's an R package issue it's unfortunate to read that we have a backward compatibility issue. Also note: We already have a solution and we'll verify this fix and distribute it as part of 3.1.1
  • The problem around dialogs that are supposed to be opened during a node's execution. That's borderline as this has always been fishy. We'll try to address it but it might be rserve just doesn't support it (Jonathan is currently looking into it.) That has never been the intended use of the software.

Hope this puts things into perspective. This change was not mentioned prominently in the what's-new page or the corresponding news article because "in theory" the change is minor from the user perspective. Except that it apparently isn't. If there are other problems coming up around the R nodes not properly working and breaking backward compatibility these will be treated with highest priority.

On the plus side: One of our KNIME customers who was sponsoring this change has already switched to using 3.1 and they are very happy to see their R scripts running in parallel now.

1 Like

Yes, running R scripts in parallel is definitely worth (in my opinion) any problems associated with the switch.

Thanks a lot for the information.

-Brian

Hi everyone,

when I run an R node on Knime 3.1, it fails with

.../Rserve.dbg: error while loading shared libraries: libR.so: cannot open shared object file: No such file or directory

For Knime < 3.1, setting LD_LIBRARY_PATH accordingly had helped, but not anymore.

Edit: One more thing: the LD_LIBRARY_PATH setting is not necessary for running Rserve from an R session launched from the console.

Just published a couple of fixes via the online update site without making it an official KNIME Analytics Platform 3.1.1 release. This should fix the problems mentioned above. Just run File -> Update KNIME...

Setting symlinks to the libR.so didn't help either. I tried to link libR.so from

my home directory, Knime working directory, in R-3.2.2/ , directory of the R executable: R-3.2.2/lib64/R/ , and the directory of Rserve.dbg: R-3.2.2/lib64/R/library/Rserve/libs/

Hey KNIMEers,

I'm not sure if this deserves a unique thread, but I am also having problems post 3.1 with the R Extension.  I am unablbe to produce a view from the R View (Table) node using the script plot(iris).  When I do so, I get the stack below in my KNIME Log.  

Any thoughts?

Thanks in advance and keep up the good work.

java.lang.RuntimeException: No Image was created by the R-Script

    at org.knime.r.RViewNodeModel.postExecuteInternal(RViewNodeModel.java:115)

    at org.knime.r.RViewNodeModel.execute(RViewNodeModel.java:102)

    at org.knime.core.node.NodeModel.executeModel(NodeModel.java:563)

    at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1141)

    at org.knime.core.node.Node.execute(Node.java:933)

    at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:556)

    at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)

    at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)

    at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)

    at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)

    at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

    at java.util.concurrent.FutureTask.run(FutureTask.java:266)

    at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)

    at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

 

Hi Aaron,

I have no problem with execution with this version on Windows:   KNIME Interactive R Statistics Integration    3.1.0.v201512031416    org.knime.features.r.feature.group    KNIME GmbH, Konstanz, Germany

(Though in the configuration dialog the Show Plot button brings up a dialog with <no plot>, even after executing the script.)

Cheers, gabor

FWIW I am on a Mac, OSX 10.10.5 with R 3.2.1.

Hi Aaron!

On Mac you need to have the Cairo package installed in your R installation. Also, you might want to try updating the R extensions (if you haven't done so since december).

Tell me if that helped.

Regards, Jonathan.

See below for the details of my current installation. 

 

#My KNIME Versions
KNIME Analytics Platform 3.1.1.v201512151318
KNIME Interactive R Statistics Integration 3.1.1.v201512111636


# R Session Info
> sessionInfo()

R version 3.2.1 (2015-06-18)

Platform: x86_64-apple-darwin13.4.0 (64-bit)

locale:

[1] C

attached base packages:

[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Cairo_1.5-9

#The Stack


java.lang.RuntimeException: No Image was created by the R-Script
at org.knime.r.RViewNodeModel.postExecuteInternal(RViewNodeModel.java:115)
at org.knime.r.RViewNodeModel.execute(RViewNodeModel.java:102)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:563)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1141)
at org.knime.core.node.Node.execute(Node.java:933)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:556)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

Hi Aaron, 

can you verify that executing the "R Source (Table)" node works? If not, you might be missing the Rserve package. Otherwise:

I tried to reproduce the problem, but seem to be unable to. It would be great if you could turn on DEBUG output under Preferences > KNIME > KNIME GUI, clear then console, run the node again and copy the entire output into pasteall.org or pastebin.com.

Thank you in advance, 

Regards, Jonathan.

Hi Jonathan,

I have similar problems on my Mac. (On Windows evereything is running fine). I followed your instructions and here is the console out put:

DEBUG ExecuteAction                   Creating execution job for 1 node(s)...

DEBUG NodeContainer                   R Snippet 0:1800 has new state: CONFIGURED_MARKEDFOREXEC

DEBUG NodeContainer                   R Snippet 0:1800 has new state: CONFIGURED_QUEUED

DEBUG NodeContainer                   04. Correct Aliases and Tables 0 has new state: EXECUTING

DEBUG NodeContainer                   ROOT  has new state: EXECUTING

DEBUG R Snippet            0:1800     R Snippet 0:1800 doBeforePreExecution

DEBUG R Snippet            0:1800     R Snippet 0:1800 has new state: PREEXECUTE

DEBUG R Snippet            0:1800     R Snippet 0:1800 doBeforeExecution

DEBUG R Snippet            0:1800     R Snippet 0:1800 has new state: EXECUTING

DEBUG R Snippet            0:1800     Adding handler e8f02c63-1bc0-4d26-8254-fccaf8a720d5 (R Snippet 0:1800: <no directory>) - 18 in total

DEBUG R Snippet            0:1800     R Snippet 0:1800 Start execute

DEBUG RBinUtil                        External Rscript process output: 

DEBUG RBinUtil                        External Rscript process error output: Warnmeldung:'memory.limit()' is Windows-specific 

DEBUG R Snippet            0:1800     reset

ERROR R Snippet            0:1800     Execute failed: Error in R code: "Error: unable to start data editor"

DEBUG R Snippet            0:1800     Execute failed: Error in R code: "Error: unable to start data editor"

org.knime.r.controller.IRController$RException: Error in R code: "Error: unable to start data editor"

    at org.knime.r.RSnippetNodeModel.runRScript(RSnippetNodeModel.java:269)

    at org.knime.r.RSnippetNodeModel.executeSnippet(RSnippetNodeModel.java:209)

    at org.knime.r.RSnippetNodeModel.executeInternal(RSnippetNodeModel.java:162)

    at org.knime.r.RSnippetNodeModel.execute(RSnippetNodeModel.java:149)

    at org.knime.core.node.NodeModel.executeModel(NodeModel.java:563)

    at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1141)

    at org.knime.core.node.Node.execute(Node.java:933)

    at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:556)

    at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)

    at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)

    at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)

    at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)

    at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

    at java.util.concurrent.FutureTask.run(FutureTask.java:266)

    at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)

    at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

DEBUG R Snippet            0:1800     R Snippet 0:1800 doBeforePostExecution

DEBUG R Snippet            0:1800     R Snippet 0:1800 has new state: POSTEXECUTE

DEBUG R Snippet            0:1800     R Snippet 0:1800 doAfterExecute - failure

DEBUG R Snippet            0:1800     reset

DEBUG R Snippet            0:1800     clean output ports.

DEBUG R Snippet            0:1800     Removing handler e8f02c63-1bc0-4d26-8254-fccaf8a720d5 (R Snippet 0:1800: <no directory>) - 17 remaining

DEBUG R Snippet            0:1800     R Snippet 0:1800 has new state: IDLE

DEBUG R Snippet            0:1800     Configure succeeded. (R Snippet)

DEBUG R Snippet            0:1800     R Snippet 0:1800 has new state: CONFIGURED

DEBUG R Snippet            0:1800     04. Correct Aliases and Tables 0 has new state: IDLE

DEBUG NodeContainer                   ROOT  has new state: IDLE

DEBUG RBinUtil                        External Rscript process error output: Warnmeldung:'memory.limit()' is Windows-specific 

DEBUG RBinUtil                        External Rscript process output: 

DEBUG RBinUtil                        External Rscript process output: 

DEBUG RBinUtil                        External Rscript process error output: Warnmeldung:'memory.limit()' is Windows-specific

Best regards

Jerry