Maximal number of cells to suit Xmx256g RAM

Dear all,

i wrote KNIME code to normalize huge single cell data. This took me very long time. I increased now my RAM to Xmx256g and the number of cells KNIME keeps in memory to 1million.
The speed is increased, however, still only half of the RAM is used shown by the heap status.
I am wondering what the maximal or optimal cell number suit to the RAM 256g. Can somebody help me with this?

Best
Shu

The limitation in your case most likley will be disk IO. If the transformation you are doing are supported by the streaming execution (things like row or columns filters, eg transformation that do not require the whole data) then I would really look into this.

If you have say 5 such transformation in a row you can make a components and set it to streaming execution. This will then save the result/changes to disk only once and not after every node.

Also your disk speed matters. All the RAM will be meaningless if you are writing to a harddrive instead of an ssd.

1 Like

Hi @ShuLiu,
I use 10GB of memory with 10M cells. My data intense WFs could make use of more RAM, but I can’t tell how much. Somebody please tell me if that’s counterproductive.
Maybe go one order of magnitude up untill KNIME hits the limit? That’s what I would do, can’t give you any actual advice on that matter. Also keep in mind that KNIME gets reluctant to use more memory when it’s near the limit. Read that somewhere in the forums/FAQ/don’t remember.

1 Like

If you use 10M with 10GB, Maybe I need to increase my cells to 100M, I use here 256GB!! I will try this.

If it doesn’t do anything, it might be that your WF has all the memory it needs. You wrote “normalise single cell data”. Does that mean everything sits in a single cell? Could be the reason you’re only at 128GB heap space, but then again 128GB would be a lot for one cell.
Maybe run several load heavy WFs in parallel to confirm? Yes, I always use my stuff at full capacity, that’s what I have it for. :smiley:

Did you try the new columnar backend (see KNIME Workbench Guide). Make sure to increase the memory of the cache to a sensible number (e.g. -Xmx32G and cache size to something much larger in case you have 256gb avail.). Should give you a considerable speed-up.

2 Likes

We have Remote Service, I used 256GB (-Xmx256g). I realized that when I change the cellmemory to 10M cells, my KNIME is very slow in open, save. I can only use the combination 256G and 1M cells? I cannt understand this.

Following are what in our knime.ini

-startup
plugins/org.eclipse.equinox.launcher_1.6.100.v20201223-0822.jar
–launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.2.100.v20210209-1541
-vm
plugins/org.knime.binary.jre.win32.x86_64_11.0.10.20210806/jre/bin/server/jvm.dll
–launcher.defaultAction
openFile
-vmargs
-server
-Dsun.java2d.d3d=false
-Dosgi.classloader.lock=classname
-XX:+UnlockDiagnosticVMOptions
-XX:+UseG1GC
-Dsun.net.client.defaultReadTimeout=0
-XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot
-Dknime.xml.disable_external_entities=true
-Xmx256g
-Dorg.eclipse.swt.browser.IEVersion=11001
-Dsun.awt.noerasebackground=true
-Dequinox.statechange.timeout=30000
-Dorg.knime.container.cellsinmemory=10000000

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.