Hi there, I am trying to make a workflow that reads lots of big xlsx files from a folder. The reading times are not ok. One file can have around 20-25 sheets with 3000 each file. By now I have 50 files and growing each day. I tried with this workflow, without success: [image] Problems: Mediocre pc. I have to work via vpn which makes the proccessing slower than normal. Any idea?

So, fast update. At the end, as usually happens, the solution was to attack a big problem making it smaller. What I ended up doing was this: [image] 1º node: I need the Read Excel Sheet Names node for the loop inside the metanodes. Metanode, where the magic happens: 1º I filter by the sheet I want. In the first case, Abanca: [image] 2º I give the indication via the loop to the Excel reader of what I need it to read. I find this step to be highly important. It has to read “file”, not “files in folder” via the flow variable. [image] After that, the rest of the workflow is prepping. As you can see, the tedious part is the set up. I have to make one per sheet, whis is slow and manual, but once I have it ready, it is so much faster. Suggestions to automate it further would be nice! Thank you everyone for the ideas.

Fast reading

KNIME Analytics Platform

mlauber71 December 1, 2021, 11:33pm 4

@jorgemartcaam you could try and first download the files (maybe one by one) and then import them

Also there have been other debates about Excel import with big files. Useb the newest KNIME version or try R.

3 Likes