Image Reader nodes limited to 4GB for .tif files?

KNIME Extensions
1

Hello KNIP community,

I am building KNIME workflows to work with microscopy image data for biological research. The .tif file format is the most widely used in my work, and these files can often get quite large, with the current largest we’re working with around 10GB. I am finding issues trying to load large .tif files using the Image Reader nodes, and thought I would see if there is something I am not doing right.

I have tried loading the large files using each of the three Image Factories, and I get the same results, with errors saying:
WARN Image Reader (Table) 3:281 Encountered exception while reading image: Row8! view log for more info.
WARN Image Reader (Table) 3:281 Encountered errors during execution!

I have tried looking in the Error Log, and the other log that can be accessed by View > Open KNIME log, and I don’t see any additional information other than these two error message lines. I don’t think it is the heap space or a memory issue, because I am giving KNIME 20GB of RAM to work with, I’ve not seen any error indicating it is a memory issue, and when I watch the memory in my task manager I have found that KNIME just closes itself if it runs over memory by opening multiple smaller files. Smaller files that do load use more memory than the large files when they return an error, so I think the issue is not a memory problem.

I know the node can access the large files, because if I load only metadata I can access the OME-XML in the output. I have tried loading subsets of large image stacks, in multiple combinations of limiting z slices, channels, or time points, and have also tried limiting the series read in, and none of these attempts have changed the result. I have tried both the Image Reader and Image Reader (Table) nodes, and see the same behavior.

I can load the large images in Fiji/ImageJ without a problem, so I tried opening a large image inside an ImageJ1 Macro node, both with the standard open and with the Bio-Formats import, and I find that the ImageJ1 Macro node does not seem to be able to open images of any size programmatically from inside the node; I’ll make a different post about this. I also tried the IJ Image Reader (2D) – experimental, and this says it loads the large image file, but only the first plane is accessible.

It seems that the standard .tif file format specification is limited to 4GB, so I tried using a bigTiff format with a .tf2 extension, with no change in the results. I then decided to test this 4GB limit, and sure enough, a 3.93 GB image with a file size of 4.13 GB including the header will load (only with the Planar Image Factory, the others throw errors), but a 4.11 GB image with a file size of 4.31 GB will not load.

So, are the Image Reader nodes limited to 4GB for .tif files, or have I missed some option that would allow them to open the larger files? It’s not a big deal if so, I can see a workaround for our large files, and I very much appreciate all the beautiful work that has gone into KNIP that I can just use and build on, I just figured I’d see if there’s something I’m not doing right. Additionally, I find that loading the 3.93 GB image file causes KNIME to use 13 to 15 GB of RAM, according to my task manager, another thing that makes me think I may be missing an option somewhere. Does this memory usage sound correct, if so what is it doing with all that extra memory? Loading the same file in Fiji / ImageJ uses about 4.5 GB.

I look forward to any answers, thoughts, or comments. Let me know if there is any further additional info I can provide.

Thanks!

  • Jeff
1 Like
2

Hi @JeffB,

Can you elaborate a bit more about the structure of your images? I would assume that you don’t acquire a single plane of 10G, but rather multiple images planes, for instance, z slices or time points?

One thing to note here is that KNIME Analytics Platform has a built-in caching mechanism for tables or much rather table rows. That is, table rows will be cached to disk if the memory fills up too quickly. I am oversimplifying here since the JVM and the assigned heap space is also involved, etc. That’s also the reason why you should fine with processing thousands of smaller images.

I have a hunch that we are talking about images with a lot of planes. Could you confirm that?

Best,
Stefan

3

Hi Stefan,

Thanks for the reply. Yes, it is images with a lot of planes that I am working with. The 3.9 GB image I can get to load is 1600x1200 pixel image planes in 8-bit, and there are 2 channels, with 1100 time points, so 2200 planes in all. The 10GB file is similarly structured, with 2718 time points. I tested a single channel of my 10GB file, so 1600x1200 pixel image planes in 8-bit with 2718 time points, and that 5 GB file does not load. Let me know if there’s any other info I can provide.

  • Jeff
4

Could you let me know which version of KNIME Analytics Platform and Image Processing you are running? You can get that info via Help > About KNIME Analytics Platform > Installation Details: In the list of installed software you can right-click on entries and copy the version information. I am specifically looking for KNIME Analytics Platform and KNIME Image Processing.

We have fixed a bug w.r.t. lookup tables that only impacts the kind of data that you are describing. Do you have a smaller file, e.g. only 100 time points, that you’d be comfortable sharing?

Best,
Stefan

5

Hi Stefan,

Here are the requested version info entries, I tried a full program update when I saw the problem before I posted, so everything should be up to date.

KNIME Analytics Platform	4.1.3.v202005121100	org.knime.product.desktop.feature.group	KNIME AG, Zurich, Switzerland
KNIME Image Processing	1.8.1.202004280557	org.knime.knip.feature.feature.group	University of Konstanz / KNIME

Let me know if there is other info I can provide.

Here is a Google Drive link to a 100 frame file. https://drive.google.com/open?id=1dfTnxT7xqzCSdiOtch7BlI1_8AFCYsyb It is indexed with time points as z slices, because that is how Fiji / ImageJ likes to work with things. Duplicating this stack and concatenating in Fiji until it is 1150 frames, 4.1 GB, will not load in KNIME for me, while 1100 frames, 3.9 GB, will load.

Thanks for your help, let me know anything I can do on my side to help move things forward.

  • Jeff
6

Hi @JeffB,

I did some digging: ImageJ1 by default exports regular TIFF files, even if those turn out to be larger than 4G. Those files can magically still be read by ImageJ1, but since are not proper BigTIFFs not by other tools. You can replicate this issue easily by trying to import one of your large files via File > Import > Image in ImageJ/Fiji.

In order to open such files in KNIME, they have to be proper BigTIFFs. You can export them via Bio-Formats, i.e. Plugins > Bio-Formats > Bio-Formats Exporter, by using the btf file extension for your files. I am not entirely sure if BF has a check built in that would make it a BigTIFF even if the file extension were tif. This way, however, I was able to import a 6G btf file with the Image Reader.

If you want some more details, this issue has them:

Best,
Stefan

3 Likes
7

Hi Stefan,

Thanks for your reply and digging! It is always great to have some idea about why things are the way they are.

I’ll confirm that my 10 GB file fails to load in Fiji/ImageJ via File>Import>Image. If I drag and drop it on the Fiji/ImageJ toolbar it will load without a problem. I’ll also confirm that if I export it from Fiji/ImageJ via Plugins>Bio-Formats>Bio-Formats Exporter as a .btf extension, that I can then read this 10 GB .btf file into KNIME using the Image Reader node.

Thanks again for your efforts!

  • Jeff
2 Likes
8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.