I am brand new to KNIME (day 3) and am already a huge fan.
I am using it for data reduction. I then export a flatfile. I subsequently import the flatfile to R for further processing (a linear discriminant analysis).
In R, I often encounter memory errors such as "Cannot allocate vector of size 2.4 Gb"
Now, one thing I have noticed about KNIME is its outstandingly efficient use of memory.
I know I can make an R node in KNIME.
Will KNIME handle the memory for the R node or will I get exactly the same memory problems as I get when running in RStudio?
again, thanks for KNIME!
You will likely run into the same issues with an R node inside KNIME. I can't speak to your specific analysis but the memory errors with R analysis often arise to very large model matricies built during an R calculation. R will need to build these identical matricies regarless of how it is invoked.
Depending on your specific case, using a "sparse" version of your analysis within R (e.g. spare LDA) can minimize the memory required for analysis. It is worth considering. Ther are many "sparse" R packages available through CRAN.
I can confirm what joshuahoran wrote:
KNIME sends the R script and Data to R, it will therefore just behave like R while running the R script.