Reading images from Link is very slow

KNIME Extensions
1

Hi all,

I have a problem regarding large amounts of data. :smile:
I’ve been trying for 2 weeks to convert “only” 300.000 images that I have in the form of links to images using the Image Reader (Table) – KNIME Community Hub. In doing so, it keeps breaking or not responding…
I first divided the dataset to 100.000 images per node (it still froze) and then to 50.000 where it has only gone through once so far. When I tried to save it, KNIME froze again and stopped responding, but I see that it is loading the second node with 50.000 images in the background.

Is there any way I can speed up this process? Or I can offload the process to my GPU, which I hope will make it work faster?

Best regards,
Merve

2

Hi Merve,

I think this might be a case for the chunk loop. You can surround your image processing pipeline with it and then you can work with e.g. a chunk size of 50 images at a time.

Possibly even better: the Image Reader supports streaming (KNIME Streaming Execution (Beta) – KNIME Hub), which is a cool plugin that could lead to a huge speedup in your case (due to parallel execution). You can read up on this feature here Streaming data in KNIME | KNIME.

Kind regards
Marvin

5 Likes
3

ohh yess, Thank you @marvin.kickuth !

2 Likes
4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.