Table Writer -- Append if file exists

Hello,
The Table Writer node should have an option to append data if the file already exists, like the CSV writer node:

Thank you

5 Likes

@tobias.koetter love the idea :slight_smile:

@Snowy we’re currently thinking about adding a feature along those lines - can you maybe describe your use-case for that feature a bit? Maybe we’re already on to something similar (e.g. we’re also discussing versioning of tables).

7 Likes

Hi @christian.dietz
Yes, happily… I have a workflow where an R snippet is inside a loop. The “Loop End” node appends the results of the loop automatically. But in this instance, I need multiple tables from my R snippet, which requires multiple “R to Table” nodes. And since I can’t have two “Loop End” nodes, I need some way to collect the results of the loop – So I write to a CSV, simply for the purpose of reading it back into the same workflow for later processing/evaluation.

Here is what my workflow looks like visually:

I hope this makes sense, and maybe there is another way to solve this problem. I simply was going to use the “Table Writer” node in place of CSV because of the file write/read size/speed. It would be ideal if there was a way to collect the results of the loop without actually having to write to a file.

Hello @Snowy,

For next release dynamic loop end support is planned and is actually part of nightly build so feel free to give it a try :wink:

https://www.knime.com/nightly-build-downloads

Still append option within Table Writer node seems reasonable to me :slight_smile:

Br,
Ivan

2 Likes

Whilst this thread kind of ended more than one year ago, I just wanted to share with you two solutions for the described problem. Please check this post of mine:

@mwiegand another option would be to use the Parquet Reader and writer. This would have the benefit of preserving the column types while allowing to add data to an existing file.

Parquet has the addition benefit of working well with Big Data and R and Python.

1 Like

Thanks for the suggestion. Definitely will have a look at that :slight_smile:

1 Like