Reading 2M local JSON files - Performance Question

KNIME Analytics Platform
1

Hello,

Let’s put aside if its smart or not :wink: But I have a folder where I constantly get new JSON files dropped in. It’s a high-speed PCIE4 NVME raid so let’s assume that will not be the bottleneck. Right now I am using the list Files node to the local path and then I am using a loop to use a JSON reader, obviously, that’s nuts, but I don’t know of any node that can open multiple JSON files in parallel.

Any ideas?

2

Hi,
Maybe it is faster to use the Load Text Files node from the Vernalis extension and then the String to JSON node to parse the content.
Kind regards,
Alexander

3 Likes
3

Any feedback on performance differences would be interesting.

4

I will try that next, my current solution is that I set up a local HTTP server and then use:

  1. List files
  2. Convert them into a localhost URL
  3. Get Request to parallel read them all :wink:
5

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.