Table Writer -- Append if file exists

[Snowy]

Hello,
The Table Writer node should have an option to append data if the file already exists, like the CSV writer node:

Thank you

[christian.birkhold]

@tobias.koetter love the idea

@Snowy we’re currently thinking about adding a feature along those lines - can you maybe describe your use-case for that feature a bit? Maybe we’re already on to something similar (e.g. we’re also discussing versioning of tables).

[Snowy]

Hi @christian.dietz
Yes, happily… I have a workflow where an R snippet is inside a loop. The “Loop End” node appends the results of the loop automatically. But in this instance, I need multiple tables from my R snippet, which requires multiple “R to Table” nodes. And since I can’t have two “Loop End” nodes, I need some way to collect the results of the loop – So I write to a CSV, simply for the purpose of reading it back into the same workflow for later processing/evaluation.

Here is what my workflow looks like visually:

I hope this makes sense, and maybe there is another way to solve this problem. I simply was going to use the “Table Writer” node in place of CSV because of the file write/read size/speed. It would be ideal if there was a way to collect the results of the loop without actually having to write to a file.

[ipazin]

Hello @Snowy,

@Snowy

I hope this makes sense, and maybe there is another way to solve this problem.

For next release dynamic loop end support is planned and is actually part of nightly build so feel free to give it a try

Nightly build downloads | KNIME

Still append option within Table Writer node seems reasonable to me

Br,
Ivan

[mwiegand]

Whilst this thread kind of ended more than one year ago, I just wanted to share with you two solutions for the described problem. Please check this post of mine:

Feature Request: Table Writer Append – Display of current complexity + X

Hi, I personally find the advance of conserving column types when using the table writer node very compelling. However, the missing functionality to simply append to an existing file compared to the CSV Writer is a major bummer. As follows the workflow which display two solutions on how to append data to a table file. This has been highlighted in this thread some time ago: I’d therefore like to suggest to implement the append to file functionality for the table file type. About the “+X” …

[mlauber71]

@mwiegand another option would be to use the Parquet Reader and writer. This would have the benefit of preserving the column types while allowing to add data to an existing file.

Incremental Data Processing with Parquet – knime

Parquet has the addition benefit of working well with Big Data and R and Python.

Create a Big Data Hive/Parquet table with a partition based on an existing...

[mwiegand]

Thanks for the suggestion. Definitely will have a look at that

[MEPivnenko]

Dear Knimers, I would like to revive the topic for the upcoming 2024. This enhancement should really rock. Append is necessary for iterative workflows, for instance ELO rating calculation for a huge dataset would require n/2 Read and n/2 Write operations now. The file would grow enormous and each Read and Write would take hundreds of milliseconds. For instance the same algorithm gives a 6 times performance boost to my workflow if I use append to CSV.