Image Reader Java Array Error

#1

I’ve been trying to read SVS files as their TIFF counter part which are ~37000 by 32000 px. When trying to do this with image reader I was encountering errors at first because of how large the TIFF image is. I was able to get it to work on my Windows laptop by changing the Image Factory setting to ‘Planar Image Factory’. However, whenever I tried to make the same workflow on my Linux (Ubuntu 16.04 LTS) powerhouse, the image reader throws an error The log is attached, but it’s throwing a java error that the array is too large. Any tips on how to get this to work in Linux?

knime.log (17.7 KB)

0 Likes
#2

Hi @Astraeus121,

Try changing the reader mode to Cell Image Factory, that is the image format for very large images.

best,
Gabriel

1 Like
#3

Hey Gabriel,

So turns out this is an issue across platforms. I’ve only been able to get the SVS file to load into Knime on my laptop and using any format other than the Planar Image Factory causes it to fail. Attempts to move this to my colleges’ Windows workstations have also resulted in failures to load the SVS files. Attached is the link to a Box file containing the workflow and an example SVS file if you want to give it a try. I’ve been trying to load index 0 in the series.

Box - Knime

0 Likes
#4

Hi @Astraeus121,

An ArrayImg doesn’t work for images of this size anymore. You could, however, select a subset of the data (e.g. only the first channel of the image) to work around this issue. In the Subset Selection tab of the Image Reader, uncheck all for the Channel dimension and select which channel to open. This will allow you to open the first channel as an ArrayImg. In addition, I would suggest to manually set the Pixel type to UnsignedByteType (just to avoid any confusion during guessing of the type…).

Have they failed with the same error message from the log file that you had posted initially? Does this also happen when you only read the first channel of the image (as described above)?

I can, unfortunately, confirm that using a CellImgFactory fails for the dataset…

Best,
Stefan

0 Likes
#5

Hi @Astraeus121,
I was able to read the image with the Planar factory on linux, after setting my KNIME to use 10GB of RAM, I was able to read the image, however it took quite a while. It also failed with the Cell factory, I will take a look at why.

best,
Gabriel

0 Likes
#6

Hey @gab1one and @stelfrich,

Just a quick recap: This workflow does work on my computer but on my college’s Windows 10 machine and my linux machine an error is thrown when the Image Reader node tries to read the .svs file. In the Box file that I shared earlier there are now two knime.log files. One is for when I try to execute the image reader node on the windows machine, the other is when I try to only load 1 channel on that same windows machine. Here is a list of details that may or may not be relevant:

-All three systems are running Knime 4.0.
-I made sure that the linux and windows systems were running the same java version as my laptop as
determined by the ‘java -version’ command (JRE 8u201)
-My laptop has an intel i7 (8 logical processors, 2.8 GHz), 16 GB of RAM, and is running Windows 10 Home
-The linux machine has AMD Threadripper 2990wx (64 logical processors, 4 GHz) and 128 GB of RAM, and is running Ubuntu 16.04 LTS
-The windows machine is running an intel i9 (20 logical processors, 3.31 GHz), 64 GB of RAM, and is running Windows 10 Education
-For each one >90% of the RAM is dedicated to KNIME
-My preference is to load all of the channels because the color tiles are first being analyzed with Ilastik before further processing.
-The error is still thrown even when trying to load just one of the channels.

Thanks for your time,
Astraeus

P.S. I recognized from the node description that Cell Image Factory should be what loads an svs file this large. I’m not really bothered by how long it takes to load the imge using Planar Image Factory, I just want it to load.

0 Likes
#7

Hi @Astraeus121,

The *-channel1.log still says Array size too large: 37848 x 33661 x 3 x 1 which implies the Image Reader is still trying to read all three channels. Could you please verify the subset selection?

The Planar Image Factory doesn’t really help that much for the data that you are looking to load, i.e. large planes. The dimensionality of the 2D planes really is the issue here that should be circumvented by the Cell Image Factory, but isn’t due to some issues in the implementation. We will dig a little deeper but it will take some time.

Best,
Stefan

0 Likes