Memory problems with large image stacks?

Hi again,
I have problems if I want to use my workflow on large image stacks. Usually I was running the workflow with z-stacks of 512x512x100 (2 Channels). Everything worked fine. I'm also using a loop to process the image stacks consecutively. Now I've recorded very similar stacks where the number of z-slices was more or less doubled. If I now run the workflow I always get some strange behaviour. For example I get an error message while executing an "image feature" node that never made any problems (the error message that shows up is "Execute failed: Java heap space"). Other nodes of the workflow (which has become a bit complex I have to admit) are suddenly executed very slowly (for example a simple Sorter node takes forever). I suspected that it might be a memory problem so I set all the nodes to "write tables to disk" but the problem still persists. Now I wanted to ask if you guys know that big image stacks can cause such problems. I'm running KNIME on a Windows7 64 bit system with 6GB of RAM.

Cheers,

Christian

 

Hi Christian,

Some questions:

  • Do you have a labeling for this image stack? if so: how is it generated? Connected Component Analysis?
  • What is the pixel type of the image (ByteType, ShortType, UnsignedShortType...)? What is the dimensionality of the new image stacks?
  • Did you already increase the heap space in your knime.ini?

You are right, it seems to be a memory problem. I think it might have something to do with the labeling. The problem is, that we can only keep complete images in the mempry since now, i.e. not parts of images.  This means the complete image + labeling (if needed) which are processed need to fit into the memory. As the labeling often is stored as an integer image of the same dimensionality as the image, it gets really large. e.g. if you have a 512x512x100x2 Image, i.e. a labeling of the same size, this would be 1.6GB large. There are workarounds (e.g. set the type of the resulting labeling to short in the CCA).

But maybe I can help you in more detail with the information about the image :)

Cheers!
Christian

 

PS: We are heavily working on a solution to partially load huge images into the memory. we have a prototype for that, but it's not finished yet.

 

Thanks Christian for your reply!

Regarding your questions:

Yes I'm creating several labels for the image stack (1 for each channel plus 1 that I use for filtering). I'm also pretty sure that my workflow is not very optimal when it comes to economical memory usage. In general I read in the image stack, then split up the two channels to process them in parallel. Since I'm using 16 bit images the stacks are UnsignedShortType. I guess all this can easily create memory problems. If you want I can send you my workflow by mail together with some example stack. I didn't know that it is possible to increase the heap size in the knime.ini file. Is there anything particular that I need to know about that?

Christian

Hi Christian,

Open the knime.ini (in your KNIME directory) with the text editor of your choice and set the "-Xmx1024m" or "-Xmx512m" to, let's say, -Xmx4G or something (if you have 6 GB 4G should be fine). Here is a link to the FAQ: http://tech.knime.org/faq#q4_2 of KNIME.

One more remark: If it's not important for you to lose some information of the image (e.g if you simply measure the size of a segment and you don't measure intensity values or something) you could also convert the image down to ByteType or UnsignedByteType (Image Converter -> Normalize/Scale). Then your memory consumption decreases.

Anyway: If you want us to have a look at the workflow and give you some hints how it could be optimized regarding efficiency or simplicity you can of course send it to us :)

Have a nice evening,

Christian

 

Hi Christian

Thanks for your help. I increased the heap size and it's working now. We are considering anyway to do our analysis eventually on some workstation where memory should not be a limiting factor anymore (hopefully).

Cheers,

Christian