Memory usage with R

I am brand new to KNIME (day 3) and am already a huge fan.

I am using it for data reduction. I then export a flatfile. I subsequently import the flatfile to R for further processing (a linear discriminant analysis).

In R, I often encounter memory errors such as "Cannot allocate vector of size 2.4 Gb"

Now, one thing I have noticed about KNIME is its outstandingly efficient use of memory.


I know I can make an R node in KNIME.

Will KNIME handle the memory for the R node  or will I get exactly the same memory problems as I get when running in RStudio?

again, thanks for KNIME!




Hi Jake,

You will likely run into the same issues with an R node inside KNIME. I can't speak to your specific analysis but the memory errors with R analysis often arise to very large model matricies built during an R calculation. R will need to build these identical matricies regarless of how it is invoked.

Depending on your specific case, using a "sparse" version of your analysis within R (e.g. spare LDA) can minimize the memory required for analysis. It is worth considering. Ther are many "sparse" R packages available through CRAN.

Hi Jake,

I can confirm what joshuahoran wrote:

KNIME sends the R script and Data to R, it will therefore just behave like R while running the R script.

Kind regards,