Ilastik output file does not exist error

ilastik
#1

Hello! I am trying to use Ilastik pixel classification for segmentation. It runs just fine on the tutorial workflow (png images), however on my workflow (windows, czi images, dapi channel column selected, classification on ilastik trained with a dapi channel only tiff image) I get the following error:

WARN KNIPLogService WARNING init.py(11): UserWarning: init: Could not import tiktorch classifier
WARN KNIPLogService WARNING 2019-03-06 09:54:29,283 opConservationTracking 3612 4820 Could not find any ILP solver
WARN KNIPLogService WARNING opStructuredTracking.py(582): SyntaxWarning: assertion is always true, perhaps remove parentheses?
WARN KNIPLogService WARNING 2019-03-06 09:54:29,315 opStructuredTracking 3612 4820 Could not find any ILP solver
WARN KNIPLogService WARNING 2019-03-06 09:54:29,317 structuredTrackingWorkflow 3612 4820 Could not find any learning solver. Tracking will use flow-based solver (DPCT). Learning for tracking will be disabled!
WARN KNIPLogService WARNING stype.py(181): UserWarning: ArrayLike.isCompatible: FIXME here
WARN KNIPLogService WARNING opExportMultipageTiff.py(100): RuntimeWarning: overflow encountered in long_scalars
ERROR Ilastik Headless (Pixel classification) 3:36 [ERROR] Error while executing Ilastik.
java.lang.IllegalStateException: Error during execution: java.nio.file.InvalidPathException: Ilastik output file does not exist: D:\Users\zeiss\AppData\Local\Temp/ilastik/6efb518d-7f62-4140-8b08-122415b18093/file0_result.tiff
at org.knime.knip.ilastik.nodes.headless.IlastikHeadlessNodeModel$IlastikCellFactory.getCell(IlastikHeadlessNodeModel.java:815)
at org.knime.core.data.container.SingleCellFactory.getCells(SingleCellFactory.java:105)
at org.knime.core.data.container.RearrangeColumnsTable.calcNewCellsForRow(RearrangeColumnsTable.java:503)
at org.knime.core.data.container.RearrangeColumnsTable.calcNewColsSynchronously(RearrangeColumnsTable.java:424)
at org.knime.core.data.container.RearrangeColumnsTable.create(RearrangeColumnsTable.java:342)
at org.knime.core.node.ExecutionContext.createColumnRearrangeTable(ExecutionContext.java:371)
at org.knime.knip.ilastik.nodes.headless.IlastikHeadlessNodeModel.execute(IlastikHeadlessNodeModel.java:322)
at org.knime.core.node.NodeModel.execute(NodeModel.java:733)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1177)
at org.knime.core.node.Node.execute(Node.java:964)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:561)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: java.nio.file.InvalidPathException: Ilastik output file does not exist: D:\Users\zeiss\AppData\Local\Temp/ilastik/6efb518d-7f62-4140-8b08-122415b18093/file0_result.tiff
at org.knime.knip.ilastik.nodes.headless.IlastikHeadlessNodeModel.readImageForRow(IlastikHeadlessNodeModel.java:408)
at org.knime.knip.ilastik.nodes.headless.IlastikHeadlessNodeModel.access$0(IlastikHeadlessNodeModel.java:393)
at org.knime.knip.ilastik.nodes.headless.IlastikHeadlessNodeModel$IlastikCellFactory.getCell(IlastikHeadlessNodeModel.java:812)
… 20 more

ERROR Ilastik Headless (Pixel classification) 3:36 Execute failed: java.lang.IllegalStateException: Error during execution: java.nio.file.InvalidPathException: Ilastik output file does not exist: D:\Users\zeiss\AppData\Local\Temp/ilastik/6efb518d-7f62-4140-8b08-122415b18093/file0_result.tiff

Since it works with the tutorial workflow, I guess the “output file does not exist” is not a path issue. Could you help me out please?

Thanks,

Paula

0 Likes

#2

The beginning of your error message has some warnings about missing python libraries. Are you sure you have everything setup properly and installed all required packages for python, knime etc?

1 Like

#3

Hi @paulage,
Can you share an example picture with us?
best,
Gabriel

0 Likes

#5

@gab1one, It is definitely my image files which are generating the error. I tried with a regular 8 bit czi generated with a different instrument and on an exported tiff version of my image, and the workflow works perfectly in both cases.

I am working with whole slide czi files (pyramidal JpgXr) generated on anAxio Z1 scanner that I split into scenes (a scene is a ROI selected for scanning on a slide) and then select one series (the biggest pyramid) on the additional options of the image reader.

The files are too big to share though the forum, but I can upload them to a share folder.

Does anyone have experience with whole slide imaging that could be helpful?

I guess I can export all mi czi files into tiffs if not…

Thanks,

Paula

0 Likes

#6

Hi @paulage,
did you train your Ilastik model on the whole slide czi file ? KNIME uses a tiff file internally to communicate with Ilastik, it is quite possible that the error is caused by the miss-match between the metadata of the training image and the image sent by KNIME.

Before you do that you can try writing one of your images from KNIME Analytics Platform to disk as a tiff file and use that file to train your Ilastik model, which you can then use to segment the images.

best,
Gabriel

0 Likes

#7

@gab1one I used a tiff to train in ilastik. czi files are not compatible with Ilastik.
I got it to work, it seems the issue was the particular czi file I was using , since it was a cropped version. Some of these cropped versions (imaged subsets) don’t open completely on knime and imageJ, and even though the one I was using did open correctly on knime and ImageJ there must have been a miss-match somewhere.
I was cropping them since all images have to be the same size. I will try the tile loop to get all same size images for Ilastik pixel classification.

Thank you!

Paula

1 Like

#8

@gab1one
I am still having issues with ilastik. It works sometime but I still get a “java.nio.file.InvalidPathException: Ilastik output file does not exist” error (as shown in the first message) other times and I can’t figure out what the difference is. It always runs with a warning even when it works. It does not seem to be the image type, it goes both ways with both tiff and czi files, always trained on a tiff of the DAPI channel created with the image writer node.

Why would would ilastik need an output file form a temp folder?

I can send you the pixel classification, image and beginning of the workflow if that’s helpful.

Thanks,

Paula

0 Likes

#9

Hi @paulage,
sharing the files with us is a good idea, I will take a look at them to see if I can figure out what is going wrong.

The ilastik integration is based on writing a temporary file to disk, which is classified by ilastik and the result read back by KNIME. If that result file doesn’t exist, something must have gone wrong with the processing.

best,
Gabriel

0 Likes

#10

I see, now we have figure out what went wrong in the processing. Here’s the workflow and pixel classification. It’s not allowing me to upload the image though.
BALKO_PERU.knwf (15.7 KB)
BALKO_PERUTMA_PD1_nuclearpixelclassification.zip (144.0 KB)

0 Likes

#11

Can you try archiving your image and upload the zip file?

best,
Gabriel

0 Likes

#12

I tried compressing and zipping…

Here is a link to the image in google drive, let me know if can access it.
https://drive.google.com/open?id=1nHK0ZIGZ1QxzRvDc0WMrZih3Uhm0Iu-l

Thanks, Paula

0 Likes

#13

Hi @paulage
I am looking at this issue, having some trouble with the new ilastik not wanting to run on my computer though. I will try on a different machine tomorrow.

best,
Gabriel

0 Likes

#14

Hello @gab1one, did you get a chance to look at this?

0 Likes

#15

Hi @paulage,

I was able to reproduce this issue, I will look into debugging it during the next days.

best,
Gabriel

0 Likes

#16

Hi,

I have the same trouble when I run the Ilastik node, did you find a solution to fix this bug?

Best,

Pierre

0 Likes

#17

@PBouillon I haven’t heard back from @gab1one or anyone else form the Knime team. In the mean time I have been using other platforms or other methods for nuclear segmentation, but I want to go back to this, I’ll let you know if I figure it out. Please let me know if you do.

Paula

0 Likes

#18

I am also experiencing the same error.
I noticed that here: Caused by: java.nio.file.InvalidPathException: Ilastik output file does not exist: D:\Users\zeiss\AppData\Local\Temp/ilastik/6efb518d-7f62-4140-8b08-122415b18093/file0_result.tiff
slashes and backslashes are used at the same time in the path description. Could this cause the issue?

0 Likes

#19

@PBouillon @gab1one
I believe I found the reason for this.
It happens to me when there aren’t enough resources available. Closing all other active programs and freeing up a bit of RAM, in my case, makes the node work just fine. Open programs and do stuff in parallel and the node fails giving the same error.
The solution might be to assign better memory/cpu limits in the config dialog or simply close everything else you don’t need.

2 Likes

#20

@ukitel @PBouillon @PBouillon

Freeing up RAM by closing all other active programs did not solve the error for me, and I have up to 98257MB of memory for knime to use during execution.

I checked the path, and the D:\Users\zeiss\AppData\Local\Temp\ilastik folder is empty, so truly “Ilastik output file does not exist: D:\Users\zeiss\AppData\Local\Temp/ilastik/27e14c25-c1e5-4623-8b24-29e55d1fde7c/file0_result.tiff”

I also checked the path to ilastik installation and it’s correct. There must be something off in the ilatisk to knime communication. I even uninstalled and re-installed both knime and ilastik. I am testing a workflow that works well for a colleague on a mac.

I tried it on 2 other windows PC and can’t get passed a “Execute failed: java.io.IOException: No externalizer available with id ImageMetadataExt3” error on the previous nodes. So I don’t know if the ilastik issue reproduces.

Paula

0 Likes

#21

In the config panel for the node, you can allocate threads and memory to ilastik, the problem for me arose when the allocated memory was lower than the available one. Moreover, the node would fail more or less when the memory got full. Could you maybe have a look at these 2 things: check how allocated memory/threads in the config panel compare to the available ones and how memory usage behaves while the node is running?
Maybe this would give a better insight. Also, how large are the images that the node is going to process?
(reinstalling also did not work for me)

0 Likes