Hello KNIP community,
I am building KNIME workflows to work with microscopy image data for biological research. The .tif file format is the most widely used in my work, and these files can often get quite large, with the current largest we’re working with around 10GB. I am finding issues trying to load large .tif files using the Image Reader nodes, and thought I would see if there is something I am not doing right.
I have tried loading the large files using each of the three Image Factories, and I get the same results, with errors saying:
WARN Image Reader (Table) 3:281 Encountered exception while reading image: Row8! view log for more info.
WARN Image Reader (Table) 3:281 Encountered errors during execution!
I have tried looking in the Error Log, and the other log that can be accessed by View > Open KNIME log, and I don’t see any additional information other than these two error message lines. I don’t think it is the heap space or a memory issue, because I am giving KNIME 20GB of RAM to work with, I’ve not seen any error indicating it is a memory issue, and when I watch the memory in my task manager I have found that KNIME just closes itself if it runs over memory by opening multiple smaller files. Smaller files that do load use more memory than the large files when they return an error, so I think the issue is not a memory problem.
I know the node can access the large files, because if I load only metadata I can access the OME-XML in the output. I have tried loading subsets of large image stacks, in multiple combinations of limiting z slices, channels, or time points, and have also tried limiting the series read in, and none of these attempts have changed the result. I have tried both the Image Reader and Image Reader (Table) nodes, and see the same behavior.
I can load the large images in Fiji/ImageJ without a problem, so I tried opening a large image inside an ImageJ1 Macro node, both with the standard open and with the Bio-Formats import, and I find that the ImageJ1 Macro node does not seem to be able to open images of any size programmatically from inside the node; I’ll make a different post about this. I also tried the IJ Image Reader (2D) – experimental, and this says it loads the large image file, but only the first plane is accessible.
It seems that the standard .tif file format specification is limited to 4GB, so I tried using a bigTiff format with a .tf2 extension, with no change in the results. I then decided to test this 4GB limit, and sure enough, a 3.93 GB image with a file size of 4.13 GB including the header will load (only with the Planar Image Factory, the others throw errors), but a 4.11 GB image with a file size of 4.31 GB will not load.
So, are the Image Reader nodes limited to 4GB for .tif files, or have I missed some option that would allow them to open the larger files? It’s not a big deal if so, I can see a workaround for our large files, and I very much appreciate all the beautiful work that has gone into KNIP that I can just use and build on, I just figured I’d see if there’s something I’m not doing right. Additionally, I find that loading the 3.93 GB image file causes KNIME to use 13 to 15 GB of RAM, according to my task manager, another thing that makes me think I may be missing an option somewhere. Does this memory usage sound correct, if so what is it doing with all that extra memory? Loading the same file in Fiji / ImageJ uses about 4.5 GB.
I look forward to any answers, thoughts, or comments. Let me know if there is any further additional info I can provide.