New R integration for 3.1

bmuchmore · December 7, 2015, 8:11pm

Hi,

I assume this information will eventually come out, but can the community get more information about the recent changes to KNIME 3.1 and R? I guess Rserve has replaced rJava, and the reason was so that multiple R nodes could be run in parallel? Do we no longer need rJava and to play with the knime.ini file to get everything working? Will the structural differences between using Rserve and rJava have any obvious effects we should be aware of? Anything else?

Any details would be appreciated. Thanks.

-Brian Muchmore

wiswedel · December 8, 2015, 3:32pm

Hi Brian,

Yes, we changed the library that enables communication between KNIME and R. Previoulsly this was rJava and is now RServe. This was motivated by some shortcommings of the rJava based integration: this was mostly around global locking preventing parallel execution of independent R nodes and some scalability problems (like R AND KNIME crashing if you feed it with too many data; or the entire data copied twice to disk to get it from one system to the other). We've also been in contact with the developers owning the rJava and RServe packages at the R project. They motivated us using RServe.

In theory that wouldn't have been a significant change as both of these libraries follow the same interface (REngine), except for some trivial Rserve book keeping of instances to start and shutdown. Unfortunately that's only the theory as we learn now that 3.1.0 is out. We are aware of the following problems:

Bad error message in case RServe is not installed into R (not a problem on most systems, specifically Windows as 'our' R binaries contain this package). Clearly a bug and will be fixed soon.
The problems around device printing as discussed here. Even though they seem to conclude it's an R package issue it's unfortunate to read that we have a backward compatibility issue. Also note: We already have a solution and we'll verify this fix and distribute it as part of 3.1.1
The problem around dialogs that are supposed to be opened during a node's execution. That's borderline as this has always been fishy. We'll try to address it but it might be rserve just doesn't support it (Jonathan is currently looking into it.) That has never been the intended use of the software.

Hope this puts things into perspective. This change was not mentioned prominently in the what's-new page or the corresponding news article because "in theory" the change is minor from the user perspective. Except that it apparently isn't. If there are other problems coming up around the R nodes not properly working and breaking backward compatibility these will be treated with highest priority.

On the plus side: One of our KNIME customers who was sponsoring this change has already switched to using 3.1 and they are very happy to see their R scripts running in parallel now.

bmuchmore · December 9, 2015, 10:15am

Yes, running R scripts in parallel is definitely worth (in my opinion) any problems associated with the switch.

Thanks a lot for the information.

-Brian

lorenz · December 12, 2015, 7:02pm

Hi everyone,

when I run an R node on Knime 3.1, it fails with

.../Rserve.dbg: error while loading shared libraries: libR.so: cannot open shared object file: No such file or directory

For Knime < 3.1, setting LD_LIBRARY_PATH accordingly had helped, but not anymore.

Edit: One more thing: the LD_LIBRARY_PATH setting is not necessary for running Rserve from an R session launched from the console.

wiswedel · December 15, 2015, 6:54pm

Just published a couple of fixes via the online update site without making it an official KNIME Analytics Platform 3.1.1 release. This should fix the problems mentioned above. Just run File -> Update KNIME...

lorenz · December 17, 2015, 1:46pm

Setting symlinks to the libR.so didn't help either. I tried to link libR.so from

my home directory, Knime working directory, in R-3.2.2/ , directory of the R executable: R-3.2.2/lib64/R/ , and the directory of Rserve.dbg: R-3.2.2/lib64/R/library/Rserve/libs/

Aaron_Hart · January 6, 2016, 8:23pm

Hey KNIMEers,

I'm not sure if this deserves a unique thread, but I am also having problems post 3.1 with the R Extension. I am unablbe to produce a view from the R View (Table) node using the script plot(iris). When I do so, I get the stack below in my KNIME Log.

Any thoughts?

Thanks in advance and keep up the good work.

java.lang.RuntimeException: No Image was created by the R-Script

at org.knime.r.RViewNodeModel.postExecuteInternal(RViewNodeModel.java:115)

at org.knime.r.RViewNodeModel.execute(RViewNodeModel.java:102)

at org.knime.core.node.NodeModel.executeModel(NodeModel.java:563)

at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1141)

at org.knime.core.node.Node.execute(Node.java:933)

at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:556)

at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)

at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)

at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)

at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)

at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)

at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

aborg · January 6, 2016, 9:54pm

Hi Aaron,

I have no problem with execution with this version on Windows: KNIME Interactive R Statistics Integration 3.1.0.v201512031416 org.knime.features.r.feature.group KNIME GmbH, Konstanz, Germany

(Though in the configuration dialog the Show Plot button brings up a dialog with <no plot>, even after executing the script.)

Cheers, gabor

Aaron_Hart · January 8, 2016, 4:58pm

FWIW I am on a Mac, OSX 10.10.5 with R 3.2.1.

jonathan.hale · January 22, 2016, 12:10pm

Hi Aaron!

On Mac you need to have the Cairo package installed in your R installation. Also, you might want to try updating the R extensions (if you haven't done so since december).

Tell me if that helped.

Regards, Jonathan.

Aaron_Hart · January 26, 2016, 8:47pm

See below for the details of my current installation.

#My KNIME Versions
KNIME Analytics Platform 3.1.1.v201512151318
KNIME Interactive R Statistics Integration 3.1.1.v201512111636

# R Session Info
> sessionInfo()

R version 3.2.1 (2015-06-18)

Platform: x86_64-apple-darwin13.4.0 (64-bit)

locale:

[1] C

attached base packages:

[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] Cairo_1.5-9

#The Stack

java.lang.RuntimeException: No Image was created by the R-Script
at org.knime.r.RViewNodeModel.postExecuteInternal(RViewNodeModel.java:115)
at org.knime.r.RViewNodeModel.execute(RViewNodeModel.java:102)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:563)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1141)
at org.knime.core.node.Node.execute(Node.java:933)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:556)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

jonathan.hale · January 28, 2016, 2:55pm

Hi Aaron,

can you verify that executing the "R Source (Table)" node works? If not, you might be missing the Rserve package. Otherwise:

I tried to reproduce the problem, but seem to be unable to. It would be great if you could turn on DEBUG output under Preferences > KNIME > KNIME GUI, clear then console, run the node again and copy the entire output into pasteall.org or pastebin.com.

Thank you in advance,

Regards, Jonathan.

Jerry · January 30, 2016, 10:16am

Hi Jonathan,

I have similar problems on my Mac. (On Windows evereything is running fine). I followed your instructions and here is the console out put:

DEBUG ExecuteAction Creating execution job for 1 node(s)...

DEBUG NodeContainer R Snippet 0:1800 has new state: CONFIGURED_MARKEDFOREXEC

DEBUG NodeContainer R Snippet 0:1800 has new state: CONFIGURED_QUEUED

DEBUG NodeContainer 04. Correct Aliases and Tables 0 has new state: EXECUTING

DEBUG NodeContainer ROOT has new state: EXECUTING

DEBUG R Snippet 0:1800 R Snippet 0:1800 doBeforePreExecution

DEBUG R Snippet 0:1800 R Snippet 0:1800 has new state: PREEXECUTE

DEBUG R Snippet 0:1800 R Snippet 0:1800 doBeforeExecution

DEBUG R Snippet 0:1800 R Snippet 0:1800 has new state: EXECUTING

DEBUG R Snippet 0:1800 Adding handler e8f02c63-1bc0-4d26-8254-fccaf8a720d5 (R Snippet 0:1800: <no directory>) - 18 in total

DEBUG R Snippet 0:1800 R Snippet 0:1800 Start execute

DEBUG RBinUtil External Rscript process output:

DEBUG RBinUtil External Rscript process error output: Warnmeldung:'memory.limit()' is Windows-specific

DEBUG R Snippet 0:1800 reset

ERROR R Snippet 0:1800 Execute failed: Error in R code: "Error: unable to start data editor"

DEBUG R Snippet 0:1800 Execute failed: Error in R code: "Error: unable to start data editor"

org.knime.r.controller.IRController$RException: Error in R code: "Error: unable to start data editor"

at org.knime.r.RSnippetNodeModel.runRScript(RSnippetNodeModel.java:269)

at org.knime.r.RSnippetNodeModel.executeSnippet(RSnippetNodeModel.java:209)

at org.knime.r.RSnippetNodeModel.executeInternal(RSnippetNodeModel.java:162)

at org.knime.r.RSnippetNodeModel.execute(RSnippetNodeModel.java:149)

at org.knime.core.node.NodeModel.executeModel(NodeModel.java:563)

at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1141)

at org.knime.core.node.Node.execute(Node.java:933)

at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:556)

at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)

at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)

at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)

at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)

at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)

at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

DEBUG R Snippet 0:1800 R Snippet 0:1800 doBeforePostExecution

DEBUG R Snippet 0:1800 R Snippet 0:1800 has new state: POSTEXECUTE

DEBUG R Snippet 0:1800 R Snippet 0:1800 doAfterExecute - failure

DEBUG R Snippet 0:1800 reset

DEBUG R Snippet 0:1800 clean output ports.

DEBUG R Snippet 0:1800 Removing handler e8f02c63-1bc0-4d26-8254-fccaf8a720d5 (R Snippet 0:1800: <no directory>) - 17 remaining

DEBUG R Snippet 0:1800 R Snippet 0:1800 has new state: IDLE

DEBUG R Snippet 0:1800 Configure succeeded. (R Snippet)

DEBUG R Snippet 0:1800 R Snippet 0:1800 has new state: CONFIGURED

DEBUG R Snippet 0:1800 04. Correct Aliases and Tables 0 has new state: IDLE

DEBUG NodeContainer ROOT has new state: IDLE

DEBUG RBinUtil External Rscript process error output: Warnmeldung:'memory.limit()' is Windows-specific

DEBUG RBinUtil External Rscript process output:

DEBUG RBinUtil External Rscript process error output: Warnmeldung:'memory.limit()' is Windows-specific

Best regards

Jerry

system · June 2, 2023, 9:31pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.