Parquet writer fails in Hub

KNIME Business Hub
1

Hello
I have a problem running a job in the Hub that should write a large Parquet type file. The node throws an error “Current workflow must be stored on a KNIME Hub” which it only brings up when the file is very large. That is, under normal conditions the flow is executed. If during the execution of the job the Memory Policy (in the editor of the running flow) is changed to “Write tables to disk”, the flow finishes its execution without problem. What happens is that the Memory Policy option in the Parquet Writer node is only visible under these conditions. That is, the Memory Policy tab is not visible in AP to leave the option “Write tables to disk” and then upload this to the Hub. What can I do?

Node in AP

image
image1337×138 17.6 KB

Node running on the Hub

image
image1025×237 66.3 KB

I really don’t undertand the difference, and I would like to solve teh error.

Thanks

2

@lsandinop have you tried splitting the file into several smaller files and upload them?

3

Hi @mlauber71
Just a question. How I can handle the issue that Parquet doesn’t support append options, and the file must be a parquet?
And the concern is that it used to work in Server but not in Hub.
Thanks

4

Well actually it does. If it is just a memory issue I think this might help. Do you run a job on the hub and what is the error message.

Are you sure you have the folders ready? How large is the file?

5

Hi @mlauber71
The error says “Unable to close resource”, and as I said it used to work in Server beacause the Knime.ini had an extension in the RAM use. And also, if we manipulate the node in the job execution changing the memory policy to “Write tables to disk” the nodes ends successfully. The File have more than 7MM of rows, and it is wrinting in a temp folder in Workflow’s data folder. The file needs to be one because is used only in excecution time. I undertand I can write several parquet files and then read them in one node, but it is something that change the way it was working on the Server and is generating discomfort among users who do not understand why, even though the Hub is better, their workflows that were running on the Server do not work.
Thanks