Download image URLs and save them as both jepeg/png file and also as a Knime Node

@badger101 I like how you keep referring to me as they/theirs, addressing my alter ego I guess :rofl:

@NeginZarbakhsh To be honest I haven’t processed so many file at once before. If your use case allows it I would stick to JPEG’s. I believe the amount of nodes that you can use with it is much higher. I’m not really sure what your intention is with the Partition. I just kept it in there for the sake of just processing a few rows, which I believe you also used it for.

To manage the processing of the images a bit more, a possible solution could be to add a Chunk Loop. This allows you to define the batch size of rows that you want to process at once. If you then run a Heavy Garbage Collector afterwards, that should help you to manage the memory issue better

It does require a small change in the Image Writer node to avoid that the filename keep overriding per each Chunk. For this I started to use the image-id column which was already included in the dataset (converted it from long to string with a Column Expression node).

I only applied it to the PNG flow for illustration purposes. I’d say play around with it a bit and see what kind of configuration and settings in general keep the flow running.

See WF v2:
Mapillary Image Collection JPG and PNG Writer V2.knwf (52.5 KB)

Hope this helps.

2 Likes