Export Large Table to JSON

Hello,

I have a workflow dealing with a large amount of data (>1M records) and after some analysis and manipulation of said data, I would like to export it into a JSON.

the “table to JSON” node returns the following error: “Execution Failed: -1651420388” and I was not able to find any solutions online.

i tried to increase the Java Heap Space (it even reached 40G) but still am facing the same error.

Any advice on how to approach this? noting that I am not able to make the data any smaller.

Appreciate the support.
regards,

Hello @laylaSabbouh,

and welcome to KNIME Community!

Why are you looking to export it into JSON? Are you sending those files to someone or importing it into another system/tool or something else? If there isn’t any specific reason for JSON you can use CSV Writer node.

Regarding Table to JSON error it’s hard to say. You should go into log file and see if there is more info about it. Alternatively you can share workflow example.

Br,
Ivan

Hello @ipazin,

we need to export into JSON as the data is sent to another system.

Unfortunately, the logs don’t show much info regarding the error, but I have faced something similar before when dealing with a large input file.

after some internal discussions, we thought of splitting the data into multiple JSONs and loading them into the target system. do you recommend any steps to do so in a dynamic way (based on the size of the input table)?

regards,

Probably it’s a huge array at the root level? Then …

(1) Write the contained objects row wise as JSON strings, append , at the end of each row - this way you don’t need to buffer the entire array in memory

(2) After file has been written, append [ at file start, strip , at file end and replace with ]. Use a script, e.g. Java, which allows this without having to read the entire file in memory.

Also look into the “JSON lines” format, which is basically only step (1)

1 Like

Hi @qqilihq,

I didn’t think of this approach before, thank you for that.

could you please explain further step (1) for me?

Much appreciated

I uploaded a simple example to my NodePit Space how I would do it:

(same principle but slightly different to how I explained it above :slight_smile: )

Hope that helps!

4 Likes

@laylaSabbouh Did it help?

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.