Reading 2M local JSON files - Performance Question


Let’s put aside if its smart or not :wink: But I have a folder where I constantly get new JSON files dropped in. It’s a high-speed PCIE4 NVME raid so let’s assume that will not be the bottleneck. Right now I am using the list Files node to the local path and then I am using a loop to use a JSON reader, obviously, that’s nuts, but I don’t know of any node that can open multiple JSON files in parallel.

Any ideas?

Maybe it is faster to use the Load Text Files node from the Vernalis extension and then the String to JSON node to parse the content.
Kind regards,


Any feedback on performance differences would be interesting.

I will try that next, my current solution is that I set up a local HTTP server and then use:

  1. List files
  2. Convert them into a localhost URL
  3. Get Request to parallel read them all :wink:

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.